<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
  "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">

<html xmlns="http://www.w3.org/1999/xhtml">
<head><title></title>
<link href="../style/ebook.css" type="text/css" rel="stylesheet">
</head>
<body>
<h1>API Reference</h1>
<div class="section" id="operators">
<span id="api-reference-operators"></span><h2 class="sigil_not_in_toc">Operators</h2>
<p>Operators allow for generation of certain types of tasks that become nodes in
the DAG when instantiated. All operators derive from <code class="docutils literal notranslate"><span class="pre">BaseOperator</span></code> and
inherit many attributes and methods that way. Refer to the <a class="reference internal" href="#baseoperator">BaseOperator</a>
documentation for more details.</p>
<p>There are 3 main types of operators:</p>
<ul class="simple">
<li>Operators that performs an <strong>action</strong>, or tell another system to
perform an action</li>
<li><strong>Transfer</strong> operators move data from one system to another</li>
<li><strong>Sensors</strong> are a certain type of operator that will keep running until a
certain criterion is met. Examples include a specific file landing in HDFS or
S3, a partition appearing in Hive, or a specific time of the day. Sensors
are derived from <code class="docutils literal notranslate"><span class="pre">BaseSensorOperator</span></code> and run a poke
method at a specified <code class="docutils literal notranslate"><span class="pre">poke_interval</span></code> until it returns <code class="docutils literal notranslate"><span class="pre">True</span></code>.</li>
</ul>
<div class="section" id="baseoperator">
<h3 class="sigil_not_in_toc">BaseOperator</h3>
<p>All operators are derived from <code class="docutils literal notranslate"><span class="pre">BaseOperator</span></code> and acquire much
functionality through inheritance. Since this is the core of the engine,
it&#x2019;s worth taking the time to understand the parameters of <code class="docutils literal notranslate"><span class="pre">BaseOperator</span></code>
to understand the primitive features that can be leveraged in your
DAGs.</p>

<pre>
class airflow.models.BaseOperator(task_id, owner=&apos;Airflow&apos;, email=None, email_on_retry=True, email_on_failure=True, retries=0, retry_delay=datetime.timedelta(0, 300), retry_exponential_backoff=False, max_retry_delay=None, start_date=None, end_date=None, schedule_interval=None, depends_on_past=False, wait_for_downstream=False, dag=None, params=None, default_args=None, adhoc=False, priority_weight=1, weight_rule=u&apos;downstream&apos;, queue=&apos;default&apos;, pool=None, sla=None, execution_timeout=None, on_failure_callback=None, on_success_callback=None, on_retry_callback=None, trigger_rule=u&apos;all_success&apos;, resources=None, run_as_user=None, task_concurrency=None, executor_config=None, inlets=None, outlets=None, *args, **kwargs)</pre>
<p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.utils.log.logging_mixin.LoggingMixin</span></code></p>
<p>Abstract base class for all operators. Since operators create objects that
become nodes in the dag, BaseOperator contains many recursive methods for
dag crawling behavior. To derive this class, you are expected to override
the constructor as well as the &#x2018;execute&#x2019; method.</p>
<p>Operators derived from this class should perform or trigger certain tasks
synchronously (wait for completion). Example of operators could be an
operator that runs a Pig job (PigOperator), a sensor operator that
waits for a partition to land in Hive (HiveSensorOperator), or one that
moves data from Hive to MySQL (Hive2MySqlOperator). Instances of these
operators (tasks) target specific operations, running specific scripts,
functions or data transfers.</p>
<p>This class is abstract and shouldn&#x2019;t be instantiated. Instantiating a
class derived from this one results in the creation of a task object,
which ultimately becomes a node in DAG objects. Task dependencies should
be set by using the set_upstream and/or set_downstream methods.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>task_id</strong> (<em>string</em>) &#x2013; a unique, meaningful id for the task</li>
<li><strong>owner</strong> (<em>string</em>) &#x2013; the owner of the task, using the unix username is recommended</li>
<li><strong>retries</strong> (<em>int</em>) &#x2013; the number of retries that should be performed before
failing the task</li>
<li><strong>retry_delay</strong> (<em>timedelta</em>) &#x2013; delay between retries</li>
<li><strong>retry_exponential_backoff</strong> (<em>bool</em>) &#x2013; allow progressive longer waits between
retries by using exponential backoff algorithm on retry delay (delay
will be converted into seconds)</li>
<li><strong>max_retry_delay</strong> (<em>timedelta</em>) &#x2013; maximum delay interval between retries</li>
<li><strong>start_date</strong> (<em>datetime</em>) &#x2013; The <code class="docutils literal notranslate"><span class="pre">start_date</span></code> for the task, determines
the <code class="docutils literal notranslate"><span class="pre">execution_date</span></code> for the first task instance. The best practice
is to have the start_date rounded
to your DAG&#x2019;s <code class="docutils literal notranslate"><span class="pre">schedule_interval</span></code>. Daily jobs have their start_date
some day at 00:00:00, hourly jobs have their start_date at 00:00
of a specific hour. Note that Airflow simply looks at the latest
<code class="docutils literal notranslate"><span class="pre">execution_date</span></code> and adds the <code class="docutils literal notranslate"><span class="pre">schedule_interval</span></code> to determine
the next <code class="docutils literal notranslate"><span class="pre">execution_date</span></code>. It is also very important
to note that different tasks&#x2019; dependencies
need to line up in time. If task A depends on task B and their
start_date are offset in a way that their execution_date don&#x2019;t line
up, A&#x2019;s dependencies will never be met. If you are looking to delay
a task, for example running a daily task at 2AM, look into the
<code class="docutils literal notranslate"><span class="pre">TimeSensor</span></code> and <code class="docutils literal notranslate"><span class="pre">TimeDeltaSensor</span></code>. We advise against using
dynamic <code class="docutils literal notranslate"><span class="pre">start_date</span></code> and recommend using fixed ones. Read the
FAQ entry about start_date for more information.</li>
<li><strong>end_date</strong> (<em>datetime</em>) &#x2013; if specified, the scheduler won&#x2019;t go beyond this date</li>
<li><strong>depends_on_past</strong> (<em>bool</em>) &#x2013; when set to true, task instances will run
sequentially while relying on the previous task&#x2019;s schedule to
succeed. The task instance for the start_date is allowed to run.</li>
<li><strong>wait_for_downstream</strong> (<em>bool</em>) &#x2013; when set to true, an instance of task
X will wait for tasks immediately downstream of the previous instance
of task X to finish successfully before it runs. This is useful if the
different instances of a task X alter the same asset, and this asset
is used by tasks downstream of task X. Note that depends_on_past
is forced to True wherever wait_for_downstream is used.</li>
<li><strong>queue</strong> (<em>str</em>) &#x2013; which queue to target when running this job. Not
all executors implement queue management, the CeleryExecutor
does support targeting specific queues.</li>
<li><strong>dag</strong> (<a class="reference internal" href="#airflow.models.DAG" title="airflow.models.DAG"><em>DAG</em></a>) &#x2013; a reference to the dag the task is attached to (if any)</li>
<li><strong>priority_weight</strong> (<em>int</em>) &#x2013; priority weight of this task against other task.
This allows the executor to trigger higher priority tasks before
others when things get backed up.</li>
<li><strong>weight_rule</strong> (<em>str</em>) &#x2013; weighting method used for the effective total
priority weight of the task. Options are:
<code class="docutils literal notranslate"><span class="pre">{</span> <span class="pre">downstream</span> <span class="pre">|</span> <span class="pre">upstream</span> <span class="pre">|</span> <span class="pre">absolute</span> <span class="pre">}</span></code> default is <code class="docutils literal notranslate"><span class="pre">downstream</span></code>
When set to <code class="docutils literal notranslate"><span class="pre">downstream</span></code> the effective weight of the task is the
aggregate sum of all downstream descendants. As a result, upstream
tasks will have higher weight and will be scheduled more aggressively
when using positive weight values. This is useful when you have
multiple dag run instances and desire to have all upstream tasks to
complete for all runs before each dag can continue processing
downstream tasks. When set to <code class="docutils literal notranslate"><span class="pre">upstream</span></code> the effective weight is the
aggregate sum of all upstream ancestors. This is the opposite where
downtream tasks have higher weight and will be scheduled more
aggressively when using positive weight values. This is useful when you
have multiple dag run instances and prefer to have each dag complete
before starting upstream tasks of other dags.  When set to
<code class="docutils literal notranslate"><span class="pre">absolute</span></code>, the effective weight is the exact <code class="docutils literal notranslate"><span class="pre">priority_weight</span></code>
specified without additional weighting. You may want to do this when
you know exactly what priority weight each task should have.
Additionally, when set to <code class="docutils literal notranslate"><span class="pre">absolute</span></code>, there is bonus effect of
significantly speeding up the task creation process as for very large
DAGS. Options can be set as string or using the constants defined in
the static class <code class="docutils literal notranslate"><span class="pre">airflow.utils.WeightRule</span></code></li>
<li><strong>pool</strong> (<em>str</em>) &#x2013; the slot pool this task should run in, slot pools are a
way to limit concurrency for certain tasks</li>
<li><strong>sla</strong> (<em>datetime.timedelta</em>) &#x2013; time by which the job is expected to succeed. Note that
this represents the <code class="docutils literal notranslate"><span class="pre">timedelta</span></code> after the period is closed. For
example if you set an SLA of 1 hour, the scheduler would send an email
soon after 1:00AM on the <code class="docutils literal notranslate"><span class="pre">2016-01-02</span></code> if the <code class="docutils literal notranslate"><span class="pre">2016-01-01</span></code> instance
has not succeeded yet.
The scheduler pays special attention for jobs with an SLA and
sends alert
emails for sla misses. SLA misses are also recorded in the database
for future reference. All tasks that share the same SLA time
get bundled in a single email, sent soon after that time. SLA
notification are sent once and only once for each task instance.</li>
<li><strong>execution_timeout</strong> (<em>datetime.timedelta</em>) &#x2013; max time allowed for the execution of
this task instance, if it goes beyond it will raise and fail.</li>
<li><strong>on_failure_callback</strong> (<em>callable</em>) &#x2013; a function to be called when a task instance
of this task fails. a context dictionary is passed as a single
parameter to this function. Context contains references to related
objects to the task instance and is documented under the macros
section of the API.</li>
<li><strong>on_retry_callback</strong> &#x2013; much like the <code class="docutils literal notranslate"><span class="pre">on_failure_callback</span></code> except
that it is executed when retries occur.</li>
<li><strong>on_success_callback</strong> (<em>callable</em>) &#x2013; much like the <code class="docutils literal notranslate"><span class="pre">on_failure_callback</span></code> except
that it is executed when the task succeeds.</li>
<li><strong>trigger_rule</strong> (<em>str</em>) &#x2013; defines the rule by which dependencies are applied
for the task to get triggered. Options are:
<code class="docutils literal notranslate"><span class="pre">{</span> <span class="pre">all_success</span> <span class="pre">|</span> <span class="pre">all_failed</span> <span class="pre">|</span> <span class="pre">all_done</span> <span class="pre">|</span> <span class="pre">one_success</span> <span class="pre">|</span>
<span class="pre">one_failed</span> <span class="pre">|</span> <span class="pre">dummy}</span></code>
default is <code class="docutils literal notranslate"><span class="pre">all_success</span></code>. Options can be set as string or
using the constants defined in the static class
<code class="docutils literal notranslate"><span class="pre">airflow.utils.TriggerRule</span></code></li>
<li><strong>resources</strong> (<em>dict</em>) &#x2013; A map of resource parameter names (the argument names of the
Resources constructor) to their values.</li>
<li><strong>run_as_user</strong> (<em>str</em>) &#x2013; unix username to impersonate while running the task</li>
<li><strong>task_concurrency</strong> (<em>int</em>) &#x2013; When set, a task will be able to limit the concurrent
runs across execution_dates</li>
<li><strong>executor_config</strong> (<em>dict</em>) &#x2013; <p>Additional task-level configuration parameters that are
interpreted by a specific executor. Parameters are namespaced by the name of
executor.
<a href="#id1"><span class="problematic" id="id2">``</span></a>example: to run this task in a specific docker container through
the KubernetesExecutor
MyOperator(&#x2026;,</p>
<blockquote>
<div>executor_config={
&#x201C;KubernetesExecutor&#x201D;:<blockquote>
<div>{&#x201C;image&#x201D;: &#x201C;myCustomDockerImage&#x201D;}
}</div>
</blockquote>
</div>
</blockquote>
<p>)``</p>
</li>
</ul>
</td>
</tr>
</tbody>
</table>

<pre>
clear(**kwargs)</pre>
<p>Clears the state of task instances associated with the task, following
the parameters specified.</p>




<pre>
dag</pre>
<p>Returns the Operator&#x2019;s DAG if set, otherwise raises an error</p>




<pre>
deps</pre>
<p>Returns the list of dependencies for the operator. These differ from execution
context dependencies in that they are specific to tasks and can be
extended/overridden by subclasses.</p>




<pre>
downstream_list</pre>
<p>@property: list of tasks directly downstream</p>




<pre>
execute(context)</pre>
<p>This is the main method to derive when creating an operator.
Context is the same dictionary used as when rendering jinja templates.</p>
<p>Refer to get_template_context for more context.</p>




<pre>
get_direct_relative_ids(upstream=False)</pre>
<p>Get the direct relative ids to the current task, upstream or
downstream.</p>




<pre>
get_direct_relatives(upstream=False)</pre>
<p>Get the direct relatives to the current task, upstream or
downstream.</p>




<pre>
get_flat_relative_ids(upstream=False, found_descendants=None)</pre>
<p>Get a flat list of relatives&#x2019; ids, either upstream or downstream.</p>




<pre>
get_flat_relatives(upstream=False)</pre>
<p>Get a flat list of relatives, either upstream or downstream.</p>




<pre>
get_task_instances(session, start_date=None, end_date=None)</pre>
<p>Get a set of task instance related to this task for a specific date
range.</p>




<pre>
has_dag()</pre>
<p>Returns True if the Operator has been assigned to a DAG.</p>




<pre>
on_kill()</pre>
<p>Override this method to cleanup subprocesses when a task instance
gets killed. Any use of the threading, subprocess or multiprocessing
module within an operator needs to be cleaned up or it will leave
ghost processes behind.</p>




<pre>
post_execute(context, *args, **kwargs)</pre>
<p>This hook is triggered right after self.execute() is called.
It is passed the execution context and any results returned by the
operator.</p>




<pre>
pre_execute(context, *args, **kwargs)</pre>
<p>This hook is triggered right before self.execute() is called.</p>




<pre>
prepare_template()</pre>
<p>Hook that is triggered after the templated fields get replaced
by their content. If you need your operator to alter the
content of the file before the template is rendered,
it should override this method to do so.</p>




<pre>
render_template(attr, content, context)</pre>
<p>Renders a template either from a file or directly in a field, and returns
the rendered result.</p>




<pre>
render_template_from_field(attr, content, context, jinja_env)</pre>
<p>Renders a template from a field. If the field is a string, it will
simply render the string and return the result. If it is a collection or
nested set of collections, it will traverse the structure and render
all strings in it.</p>




<pre>
run(start_date=None, end_date=None, ignore_first_depends_on_past=False, ignore_ti_state=False, mark_success=False)</pre>
<p>Run a set of task instances for a date range.</p>




<pre>
schedule_interval</pre>
<p>The schedule interval of the DAG always wins over individual tasks so
that tasks within a DAG always line up. The task still needs a
schedule_interval as it may not be attached to a DAG.</p>




<pre>
set_downstream(task_or_task_list)</pre>
<p>Set a task or a task list to be directly downstream from the current
task.</p>




<pre>
set_upstream(task_or_task_list)</pre>
<p>Set a task or a task list to be directly upstream from the current
task.</p>




<pre>
upstream_list</pre>
<p>@property: list of tasks directly upstream</p>




<pre>
xcom_pull(context, task_ids=None, dag_id=None, key=u&apos;return_value&apos;, include_prior_dates=None)</pre>
<p>See TaskInstance.xcom_pull()</p>




<pre>
xcom_push(context, key, value, execution_date=None)</pre>
<p>See TaskInstance.xcom_push()</p>






</div>
<div class="section" id="basesensoroperator">
<h3 class="sigil_not_in_toc">BaseSensorOperator</h3>
<p>All sensors are derived from <code class="docutils literal notranslate"><span class="pre">BaseSensorOperator</span></code>. All sensors inherit
the <code class="docutils literal notranslate"><span class="pre">timeout</span></code> and <code class="docutils literal notranslate"><span class="pre">poke_interval</span></code> on top of the <code class="docutils literal notranslate"><span class="pre">BaseOperator</span></code>
attributes.</p>

<pre>
class airflow.sensors.base_sensor_operator.BaseSensorOperator(poke_interval=60, timeout=604800, soft_fail=False, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a>, <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.SkipMixin</span></code></p>
<p>Sensor operators are derived from this class an inherit these attributes.</p>

<pre>Sensor operators keep executing at a time interval and succeed when</pre>
a criteria is met and fail if and when they time out.

<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>soft_fail</strong> (<em>bool</em>) &#x2013; Set to true to mark the task as SKIPPED on failure</li>
<li><strong>poke_interval</strong> (<em>int</em>) &#x2013; Time in seconds that the job should wait in
between each tries</li>
<li><strong>timeout</strong> (<em>int</em>) &#x2013; Time, in seconds before the task times out and fails.</li>
</ul>
</td>
</tr>
</tbody>
</table>

<pre>
poke(context)</pre>
<p>Function that the sensors defined while deriving this class should
override.</p>






</div>
<div class="section" id="core-operators">
<h3 class="sigil_not_in_toc">Core Operators</h3>
<div class="section" id="id3">
<h4 class="sigil_not_in_toc">Operators</h4>

<pre>
class airflow.operators.bash_operator.BashOperator(bash_command, xcom_push=False, env=None, output_encoding=&apos;utf-8&apos;, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Execute a Bash script, command or set of commands.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>bash_command</strong> (<em>string</em>) &#x2013; The command, set of commands or reference to a
bash script (must be &#x2018;.sh&#x2019;) to be executed. (templated)</li>
<li><strong>xcom_push</strong> (<em>bool</em>) &#x2013; If xcom_push is True, the last line written to stdout
will also be pushed to an XCom when the bash command completes.</li>
<li><strong>env</strong> (<em>dict</em>) &#x2013; If env is not None, it must be a mapping that defines the
environment variables for the new process; these are used instead
of inheriting the current process environment, which is the default
behavior. (templated)</li>
</ul>
</td>
</tr>
</tbody>
</table>

<pre>
execute(context)</pre>
<p>Execute the bash command in a temporary directory
which will be cleaned afterwards</p>







<pre>
class airflow.operators.python_operator.BranchPythonOperator(python_callable, op_args=None, op_kwargs=None, provide_context=False, templates_dict=None, templates_exts=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.operators.python_operator.PythonOperator" title="airflow.operators.python_operator.PythonOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.operators.python_operator.PythonOperator</span></code></a>, <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.SkipMixin</span></code></p>
<p>Allows a workflow to &#x201C;branch&#x201D; or follow a single path following the
execution of this task.</p>
<p>It derives the PythonOperator and expects a Python function that returns
the task_id to follow. The task_id returned should point to a task
directly downstream from {self}. All other &#x201C;branches&#x201D; or
directly downstream tasks are marked with a state of <code class="docutils literal notranslate"><span class="pre">skipped</span></code> so that
these paths can&#x2019;t move forward. The <code class="docutils literal notranslate"><span class="pre">skipped</span></code> states are propageted
downstream to allow for the DAG state to fill up and the DAG run&#x2019;s state
to be inferred.</p>
<p>Note that using tasks with <code class="docutils literal notranslate"><span class="pre">depends_on_past=True</span></code> downstream from
<code class="docutils literal notranslate"><span class="pre">BranchPythonOperator</span></code> is logically unsound as <code class="docutils literal notranslate"><span class="pre">skipped</span></code> status
will invariably lead to block tasks that depend on their past successes.
<code class="docutils literal notranslate"><span class="pre">skipped</span></code> states propagates where all directly upstream tasks are
<code class="docutils literal notranslate"><span class="pre">skipped</span></code>.</p>




<pre>
class airflow.operators.check_operator.CheckOperator(sql, conn_id=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Performs checks against a db. The <code class="docutils literal notranslate"><span class="pre">CheckOperator</span></code> expects
a sql query that will return a single row. Each value on that
first row is evaluated using python <code class="docutils literal notranslate"><span class="pre">bool</span></code> casting. If any of the
values return <code class="docutils literal notranslate"><span class="pre">False</span></code> the check is failed and errors out.</p>
<p>Note that Python bool casting evals the following as <code class="docutils literal notranslate"><span class="pre">False</span></code>:</p>
<ul class="simple">
<li><code class="docutils literal notranslate"><span class="pre">False</span></code></li>
<li><code class="docutils literal notranslate"><span class="pre">0</span></code></li>
<li>Empty string (<code class="docutils literal notranslate"><span class="pre">&quot;&quot;</span></code>)</li>
<li>Empty list (<code class="docutils literal notranslate"><span class="pre">[]</span></code>)</li>
<li>Empty dictionary or set (<code class="docutils literal notranslate"><span class="pre">{}</span></code>)</li>
</ul>
<p>Given a query like <code class="docutils literal notranslate"><span class="pre">SELECT</span> <span class="pre">COUNT(*)</span> <span class="pre">FROM</span> <span class="pre">foo</span></code>, it will fail only if
the count <code class="docutils literal notranslate"><span class="pre">==</span> <span class="pre">0</span></code>. You can craft much more complex query that could,
for instance, check that the table has the same number of rows as
the source table upstream, or that the count of today&#x2019;s partition is
greater than yesterday&#x2019;s partition, or that a set of metrics are less
than 3 standard deviation for the 7 day average.</p>
<p>This operator can be used as a data quality check in your pipeline, and
depending on where you put it in your DAG, you have the choice to
stop the critical path, preventing from
publishing dubious data, or on the side and receive email alerts
without stopping the progress of the DAG.</p>
<p>Note that this is an abstract class and get_db_hook
needs to be defined. Whereas a get_db_hook is hook that gets a
single record from an external source.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><strong>sql</strong> (<em>string</em>) &#x2013; the sql to be executed. (templated)</td>
</tr>
</tbody>
</table>




<pre>
class airflow.operators.docker_operator.DockerOperator(image, api_version=None, command=None, cpus=1.0, docker_url=&apos;unix://var/run/docker.sock&apos;, environment=None, force_pull=False, mem_limit=None, network_mode=None, tls_ca_cert=None, tls_client_cert=None, tls_client_key=None, tls_hostname=None, tls_ssl_version=None, tmp_dir=&apos;/tmp/airflow&apos;, user=None, volumes=None, working_dir=None, xcom_push=False, xcom_all=False, docker_conn_id=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Execute a command inside a docker container.</p>
<p>A temporary directory is created on the host and
mounted into a container to allow storing files
that together exceed the default disk size of 10GB in a container.
The path to the mounted directory can be accessed
via the environment variable <code class="docutils literal notranslate"><span class="pre">AIRFLOW_TMP_DIR</span></code>.</p>
<p>If a login to a private registry is required prior to pulling the image, a
Docker connection needs to be configured in Airflow and the connection ID
be provided with the parameter <code class="docutils literal notranslate"><span class="pre">docker_conn_id</span></code>.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>image</strong> (<em>str</em>) &#x2013; Docker image from which to create the container.</li>
<li><strong>api_version</strong> (<em>str</em>) &#x2013; Remote API version. Set to <code class="docutils literal notranslate"><span class="pre">auto</span></code> to automatically
detect the server&#x2019;s version.</li>
<li><strong>command</strong> (<em>str</em><em> or </em><em>list</em>) &#x2013; Command to be run in the container. (templated)</li>
<li><strong>cpus</strong> (<em>float</em>) &#x2013; Number of CPUs to assign to the container.
This value gets multiplied with 1024. See
<a class="reference external" href="https://docs.docker.com/engine/reference/run/#cpu-share-constraint">https://docs.docker.com/engine/reference/run/#cpu-share-constraint</a></li>
<li><strong>docker_url</strong> (<em>str</em>) &#x2013; URL of the host running the docker daemon.
Default is unix://var/run/docker.sock</li>
<li><strong>environment</strong> (<em>dict</em>) &#x2013; Environment variables to set in the container. (templated)</li>
<li><strong>force_pull</strong> (<em>bool</em>) &#x2013; Pull the docker image on every run. Default is false.</li>
<li><strong>mem_limit</strong> (<em>float</em><em> or </em><em>str</em>) &#x2013; Maximum amount of memory the container can use.
Either a float value, which represents the limit in bytes,
or a string like <code class="docutils literal notranslate"><span class="pre">128m</span></code> or <code class="docutils literal notranslate"><span class="pre">1g</span></code>.</li>
<li><strong>network_mode</strong> (<em>str</em>) &#x2013; Network mode for the container.</li>
<li><strong>tls_ca_cert</strong> (<em>str</em>) &#x2013; Path to a PEM-encoded certificate authority
to secure the docker connection.</li>
<li><strong>tls_client_cert</strong> (<em>str</em>) &#x2013; Path to the PEM-encoded certificate
used to authenticate docker client.</li>
<li><strong>tls_client_key</strong> (<em>str</em>) &#x2013; Path to the PEM-encoded key used to authenticate docker client.</li>
<li><strong>tls_hostname</strong> (<em>str</em><em> or </em><em>bool</em>) &#x2013; Hostname to match against
the docker server certificate or False to disable the check.</li>
<li><strong>tls_ssl_version</strong> (<em>str</em>) &#x2013; Version of SSL to use when communicating with docker daemon.</li>
<li><strong>tmp_dir</strong> (<em>str</em>) &#x2013; Mount point inside the container to
a temporary directory created on the host by the operator.
The path is also made available via the environment variable
<code class="docutils literal notranslate"><span class="pre">AIRFLOW_TMP_DIR</span></code> inside the container.</li>
<li><strong>user</strong> (<em>int</em><em> or </em><em>str</em>) &#x2013; Default user inside the docker container.</li>
<li><strong>volumes</strong> &#x2013; List of volumes to mount into the container, e.g.
<code class="docutils literal notranslate"><span class="pre">[&apos;/host/path:/container/path&apos;,</span> <span class="pre">&apos;/host/path2:/container/path2:ro&apos;]</span></code>.</li>
<li><strong>working_dir</strong> (<em>str</em>) &#x2013; Working directory to
set on the container (equivalent to the -w switch the docker client)</li>
<li><strong>xcom_push</strong> (<em>bool</em>) &#x2013; Does the stdout will be pushed to the next step using XCom.
The default is False.</li>
<li><strong>xcom_all</strong> (<em>bool</em>) &#x2013; Push all the stdout or just the last line.
The default is False (last line).</li>
<li><strong>docker_conn_id</strong> (<em>str</em>) &#x2013; ID of the Airflow connection to use</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.operators.dummy_operator.DummyOperator(*args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Operator that does literally nothing. It can be used to group tasks in a
DAG.</p>




<pre>
class airflow.operators.druid_check_operator.DruidCheckOperator(sql, druid_broker_conn_id=&apos;druid_broker_default&apos;, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.operators.check_operator.CheckOperator" title="airflow.operators.check_operator.CheckOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.operators.check_operator.CheckOperator</span></code></a></p>
<p>Performs checks against Druid. The <code class="docutils literal notranslate"><span class="pre">DruidCheckOperator</span></code> expects
a sql query that will return a single row. Each value on that
first row is evaluated using python <code class="docutils literal notranslate"><span class="pre">bool</span></code> casting. If any of the
values return <code class="docutils literal notranslate"><span class="pre">False</span></code> the check is failed and errors out.</p>
<p>Note that Python bool casting evals the following as <code class="docutils literal notranslate"><span class="pre">False</span></code>:</p>
<ul class="simple">
<li><code class="docutils literal notranslate"><span class="pre">False</span></code></li>
<li><code class="docutils literal notranslate"><span class="pre">0</span></code></li>
<li>Empty string (<code class="docutils literal notranslate"><span class="pre">&quot;&quot;</span></code>)</li>
<li>Empty list (<code class="docutils literal notranslate"><span class="pre">[]</span></code>)</li>
<li>Empty dictionary or set (<code class="docutils literal notranslate"><span class="pre">{}</span></code>)</li>
</ul>
<p>Given a query like <code class="docutils literal notranslate"><span class="pre">SELECT</span> <span class="pre">COUNT(*)</span> <span class="pre">FROM</span> <span class="pre">foo</span></code>, it will fail only if
the count <code class="docutils literal notranslate"><span class="pre">==</span> <span class="pre">0</span></code>. You can craft much more complex query that could,
for instance, check that the table has the same number of rows as
the source table upstream, or that the count of today&#x2019;s partition is
greater than yesterday&#x2019;s partition, or that a set of metrics are less
than 3 standard deviation for the 7 day average.
This operator can be used as a data quality check in your pipeline, and
depending on where you put it in your DAG, you have the choice to
stop the critical path, preventing from
publishing dubious data, or on the side and receive email alterts
without stopping the progress of the DAG.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>sql</strong> (<em>string</em>) &#x2013; the sql to be executed</li>
<li><strong>druid_broker_conn_id</strong> (<em>string</em>) &#x2013; reference to the druid broker</li>
</ul>
</td>
</tr>
</tbody>
</table>

<pre>
get_db_hook()</pre>
<p>Return the druid db api hook.</p>




<pre>
get_first(sql)</pre>
<p>Executes the druid sql to druid broker and returns the first resulting row.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><strong>sql</strong> (<em>str</em>) &#x2013; the sql statement to be executed (str)</td>
</tr>
</tbody>
</table>







<pre>
class airflow.operators.email_operator.EmailOperator(to, subject, html_content, files=None, cc=None, bcc=None, mime_subtype=&apos;mixed&apos;, mime_charset=&apos;us_ascii&apos;, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Sends an email.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>to</strong> (<em>list</em><em> or </em><em>string</em><em> (</em><em>comma</em><em> or </em><em>semicolon delimited</em><em>)</em>) &#x2013; list of emails to send the email to. (templated)</li>
<li><strong>subject</strong> (<em>string</em>) &#x2013; subject line for the email. (templated)</li>
<li><strong>html_content</strong> (<em>string</em>) &#x2013; content of the email, html markup
is allowed. (templated)</li>
<li><strong>files</strong> (<em>list</em>) &#x2013; file names to attach in email</li>
<li><strong>cc</strong> (<em>list</em><em> or </em><em>string</em><em> (</em><em>comma</em><em> or </em><em>semicolon delimited</em><em>)</em>) &#x2013; list of recipients to be added in CC field</li>
<li><strong>bcc</strong> (<em>list</em><em> or </em><em>string</em><em> (</em><em>comma</em><em> or </em><em>semicolon delimited</em><em>)</em>) &#x2013; list of recipients to be added in BCC field</li>
<li><strong>mime_subtype</strong> (<em>string</em>) &#x2013; MIME sub content type</li>
<li><strong>mime_charset</strong> (<em>string</em>) &#x2013; character set parameter added to the Content-Type
header.</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.operators.generic_transfer.GenericTransfer(sql, destination_table, source_conn_id, destination_conn_id, preoperator=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Moves data from a connection to another, assuming that they both
provide the required methods in their respective hooks. The source hook
needs to expose a <cite>get_records</cite> method, and the destination a
<cite>insert_rows</cite> method.</p>
<p>This is meant to be used on small-ish datasets that fit in memory.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>sql</strong> (<em>str</em>) &#x2013; SQL query to execute against the source database. (templated)</li>
<li><strong>destination_table</strong> (<em>str</em>) &#x2013; target table. (templated)</li>
<li><strong>source_conn_id</strong> (<em>str</em>) &#x2013; source connection</li>
<li><strong>destination_conn_id</strong> (<em>str</em>) &#x2013; source connection</li>
<li><strong>preoperator</strong> (<em>str</em><em> or </em><em>list of str</em>) &#x2013; sql statement or list of statements to be
executed prior to loading the data. (templated)</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.operators.hive_to_druid.HiveToDruidTransfer(sql, druid_datasource, ts_dim, metric_spec=None, hive_cli_conn_id=&apos;hive_cli_default&apos;, druid_ingest_conn_id=&apos;druid_ingest_default&apos;, metastore_conn_id=&apos;metastore_default&apos;, hadoop_dependency_coordinates=None, intervals=None, num_shards=-1, target_partition_size=-1, query_granularity=&apos;NONE&apos;, segment_granularity=&apos;DAY&apos;, hive_tblproperties=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Moves data from Hive to Druid, [del]note that for now the data is loaded
into memory before being pushed to Druid, so this operator should
be used for smallish amount of data.[/del]</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>sql</strong> (<em>str</em>) &#x2013; SQL query to execute against the Druid database. (templated)</li>
<li><strong>druid_datasource</strong> (<em>str</em>) &#x2013; the datasource you want to ingest into in druid</li>
<li><strong>ts_dim</strong> (<em>str</em>) &#x2013; the timestamp dimension</li>
<li><strong>metric_spec</strong> (<em>list</em>) &#x2013; the metrics you want to define for your data</li>
<li><strong>hive_cli_conn_id</strong> (<em>str</em>) &#x2013; the hive connection id</li>
<li><strong>druid_ingest_conn_id</strong> (<em>str</em>) &#x2013; the druid ingest connection id</li>
<li><strong>metastore_conn_id</strong> (<em>str</em>) &#x2013; the metastore connection id</li>
<li><strong>hadoop_dependency_coordinates</strong> (<em>list of str</em>) &#x2013; list of coordinates to squeeze
int the ingest json</li>
<li><strong>intervals</strong> (<em>list</em>) &#x2013; list of time intervals that defines segments,
this is passed as is to the json object. (templated)</li>
<li><strong>hive_tblproperties</strong> (<em>dict</em>) &#x2013; additional properties for tblproperties in
hive for the staging table</li>
</ul>
</td>
</tr>
</tbody>
</table>

<pre>
construct_ingest_query(static_path, columns)</pre>
<p>Builds an ingest query for an HDFS TSV load.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>static_path</strong> (<em>str</em>) &#x2013; The path on hdfs where the data is</li>
<li><strong>columns</strong> (<em>list</em>) &#x2013; List of all the columns that are available</li>
</ul>
</td>
</tr>
</tbody>
</table>







<pre>
class airflow.operators.hive_to_mysql.HiveToMySqlTransfer(sql, mysql_table, hiveserver2_conn_id=&apos;hiveserver2_default&apos;, mysql_conn_id=&apos;mysql_default&apos;, mysql_preoperator=None, mysql_postoperator=None, bulk_load=False, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Moves data from Hive to MySQL, note that for now the data is loaded
into memory before being pushed to MySQL, so this operator should
be used for smallish amount of data.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>sql</strong> (<em>str</em>) &#x2013; SQL query to execute against Hive server. (templated)</li>
<li><strong>mysql_table</strong> (<em>str</em>) &#x2013; target MySQL table, use dot notation to target a
specific database. (templated)</li>
<li><strong>mysql_conn_id</strong> (<em>str</em>) &#x2013; source mysql connection</li>
<li><strong>hiveserver2_conn_id</strong> (<em>str</em>) &#x2013; destination hive connection</li>
<li><strong>mysql_preoperator</strong> (<em>str</em>) &#x2013; sql statement to run against mysql prior to
import, typically use to truncate of delete in place
of the data coming in, allowing the task to be idempotent (running
the task twice won&#x2019;t double load data). (templated)</li>
<li><strong>mysql_postoperator</strong> (<em>str</em>) &#x2013; sql statement to run against mysql after the
import, typically used to move data from staging to
production and issue cleanup commands. (templated)</li>
<li><strong>bulk_load</strong> (<em>bool</em>) &#x2013; flag to use bulk_load option.  This loads mysql directly
from a tab-delimited text file using the LOAD DATA LOCAL INFILE command.
This option requires an extra connection parameter for the
destination MySQL connection: {&#x2018;local_infile&#x2019;: true}.</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.operators.hive_to_samba_operator.Hive2SambaOperator(hql, destination_filepath, samba_conn_id=&apos;samba_default&apos;, hiveserver2_conn_id=&apos;hiveserver2_default&apos;, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Executes hql code in a specific Hive database and loads the
results of the query as a csv to a Samba location.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>hql</strong> (<em>string</em>) &#x2013; the hql to be exported. (templated)</li>
<li><strong>hiveserver2_conn_id</strong> (<em>string</em>) &#x2013; reference to the hiveserver2 service</li>
<li><strong>samba_conn_id</strong> (<em>string</em>) &#x2013; reference to the samba destination</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.operators.hive_operator.HiveOperator(hql, hive_cli_conn_id=u&apos;hive_cli_default&apos;, schema=u&apos;default&apos;, hiveconfs=None, hiveconf_jinja_translate=False, script_begin_tag=None, run_as_owner=False, mapred_queue=None, mapred_queue_priority=None, mapred_job_name=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Executes hql code or hive script in a specific Hive database.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>hql</strong> (<em>string</em>) &#x2013; the hql to be executed. Note that you may also use
a relative path from the dag file of a (template) hive
script. (templated)</li>
<li><strong>hive_cli_conn_id</strong> (<em>string</em>) &#x2013; reference to the Hive database. (templated)</li>
<li><strong>hiveconfs</strong> (<em>dict</em>) &#x2013; if defined, these key value pairs will be passed
to hive as <code class="docutils literal notranslate"><span class="pre">-hiveconf</span> <span class="pre">&quot;key&quot;=&quot;value&quot;</span></code></li>
<li><strong>hiveconf_jinja_translate</strong> (<em>boolean</em>) &#x2013; when True, hiveconf-type templating
${var} gets translated into jinja-type templating {{ var }} and
${hiveconf:var} gets translated into jinja-type templating {{ var }}.
Note that you may want to use this along with the
<code class="docutils literal notranslate"><span class="pre">DAG(user_defined_macros=myargs)</span></code> parameter. View the DAG
object documentation for more details.</li>
<li><strong>script_begin_tag</strong> (<em>str</em>) &#x2013; If defined, the operator will get rid of the
part of the script before the first occurrence of <cite>script_begin_tag</cite></li>
<li><strong>mapred_queue</strong> (<em>string</em>) &#x2013; queue used by the Hadoop CapacityScheduler. (templated)</li>
<li><strong>mapred_queue_priority</strong> (<em>string</em>) &#x2013; priority within CapacityScheduler queue.
Possible settings include: VERY_HIGH, HIGH, NORMAL, LOW, VERY_LOW</li>
<li><strong>mapred_job_name</strong> (<em>string</em>) &#x2013; This name will appear in the jobtracker.
This can make monitoring easier.</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.operators.hive_stats_operator.HiveStatsCollectionOperator(table, partition, extra_exprs=None, col_blacklist=None, assignment_func=None, metastore_conn_id=&apos;metastore_default&apos;, presto_conn_id=&apos;presto_default&apos;, mysql_conn_id=&apos;airflow_db&apos;, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Gathers partition statistics using a dynamically generated Presto
query, inserts the stats into a MySql table with this format. Stats
overwrite themselves if you rerun the same date/partition.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">CREATE</span> <span class="n">TABLE</span> <span class="n">hive_stats</span> <span class="p">(</span>
    <span class="n">ds</span> <span class="n">VARCHAR</span><span class="p">(</span><span class="mi">16</span><span class="p">),</span>
    <span class="n">table_name</span> <span class="n">VARCHAR</span><span class="p">(</span><span class="mi">500</span><span class="p">),</span>
    <span class="n">metric</span> <span class="n">VARCHAR</span><span class="p">(</span><span class="mi">200</span><span class="p">),</span>
    <span class="n">value</span> <span class="n">BIGINT</span>
<span class="p">);</span>
</pre>
</div>
</div>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>table</strong> (<em>str</em>) &#x2013; the source table, in the format <code class="docutils literal notranslate"><span class="pre">database.table_name</span></code>. (templated)</li>
<li><strong>partition</strong> (<em>dict of {col:value}</em>) &#x2013; the source partition. (templated)</li>
<li><strong>extra_exprs</strong> (<em>dict</em>) &#x2013; dict of expression to run against the table where
keys are metric names and values are Presto compatible expressions</li>
<li><strong>col_blacklist</strong> (<em>list</em>) &#x2013; list of columns to blacklist, consider
blacklisting blobs, large json columns, &#x2026;</li>
<li><strong>assignment_func</strong> (<em>function</em>) &#x2013; a function that receives a column name and
a type, and returns a dict of metric names and an Presto expressions.
If None is returned, the global defaults are applied. If an
empty dictionary is returned, no stats are computed for that
column.</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.operators.check_operator.IntervalCheckOperator(table, metrics_thresholds, date_filter_column=&apos;ds&apos;, days_back=-7, conn_id=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Checks that the values of metrics given as SQL expressions are within
a certain tolerance of the ones from days_back before.</p>
<p>Note that this is an abstract class and get_db_hook
needs to be defined. Whereas a get_db_hook is hook that gets a
single record from an external source.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>table</strong> (<em>str</em>) &#x2013; the table name</li>
<li><strong>days_back</strong> (<em>int</em>) &#x2013; number of days between ds and the ds we want to check
against. Defaults to 7 days</li>
<li><strong>metrics_threshold</strong> (<em>dict</em>) &#x2013; a dictionary of ratios indexed by metrics</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.operators.jdbc_operator.JdbcOperator(sql, jdbc_conn_id=&apos;jdbc_default&apos;, autocommit=False, parameters=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Executes sql code in a database using jdbc driver.</p>
<p>Requires jaydebeapi.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>jdbc_conn_id</strong> (<em>string</em>) &#x2013; reference to a predefined database</li>
<li><strong>sql</strong> (<em>Can receive a str representing a sql statement</em><em>,
</em><em>a list of str</em><em> (</em><em>sql statements</em><em>)</em><em>, or </em><em>reference to a template file.
Template reference are recognized by str ending in &apos;.sql&apos;</em>) &#x2013; the sql code to be executed. (templated)</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.operators.latest_only_operator.LatestOnlyOperator(task_id, owner=&apos;Airflow&apos;, email=None, email_on_retry=True, email_on_failure=True, retries=0, retry_delay=datetime.timedelta(0, 300), retry_exponential_backoff=False, max_retry_delay=None, start_date=None, end_date=None, schedule_interval=None, depends_on_past=False, wait_for_downstream=False, dag=None, params=None, default_args=None, adhoc=False, priority_weight=1, weight_rule=u&apos;downstream&apos;, queue=&apos;default&apos;, pool=None, sla=None, execution_timeout=None, on_failure_callback=None, on_success_callback=None, on_retry_callback=None, trigger_rule=u&apos;all_success&apos;, resources=None, run_as_user=None, task_concurrency=None, executor_config=None, inlets=None, outlets=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a>, <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.SkipMixin</span></code></p>
<p>Allows a workflow to skip tasks that are not running during the most
recent schedule interval.</p>
<p>If the task is run outside of the latest schedule interval, all
directly downstream tasks will be skipped.</p>




<pre>
class airflow.operators.mssql_operator.MsSqlOperator(sql, mssql_conn_id=&apos;mssql_default&apos;, parameters=None, autocommit=False, database=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Executes sql code in a specific Microsoft SQL database</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>mssql_conn_id</strong> (<em>string</em>) &#x2013; reference to a specific mssql database</li>
<li><strong>sql</strong> (<em>string</em><em> or </em><em>string pointing to a template file with .sql
extension.</em><em> (</em><em>templated</em><em>)</em>) &#x2013; the sql code to be executed</li>
<li><strong>database</strong> (<em>string</em>) &#x2013; name of database which overwrite defined one in connection</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.operators.mssql_to_hive.MsSqlToHiveTransfer(sql, hive_table, create=True, recreate=False, partition=None, delimiter=u&apos;x01&apos;, mssql_conn_id=&apos;mssql_default&apos;, hive_cli_conn_id=&apos;hive_cli_default&apos;, tblproperties=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Moves data from Microsoft SQL Server to Hive. The operator runs
your query against Microsoft SQL Server, stores the file locally
before loading it into a Hive table. If the <code class="docutils literal notranslate"><span class="pre">create</span></code> or
<code class="docutils literal notranslate"><span class="pre">recreate</span></code> arguments are set to <code class="docutils literal notranslate"><span class="pre">True</span></code>,
a <code class="docutils literal notranslate"><span class="pre">CREATE</span> <span class="pre">TABLE</span></code> and <code class="docutils literal notranslate"><span class="pre">DROP</span> <span class="pre">TABLE</span></code> statements are generated.
Hive data types are inferred from the cursor&#x2019;s metadata.
Note that the table generated in Hive uses <code class="docutils literal notranslate"><span class="pre">STORED</span> <span class="pre">AS</span> <span class="pre">textfile</span></code>
which isn&#x2019;t the most efficient serialization format. If a
large amount of data is loaded and/or if the table gets
queried considerably, you may want to use this operator only to
stage the data into a temporary table before loading it into its
final destination using a <code class="docutils literal notranslate"><span class="pre">HiveOperator</span></code>.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>sql</strong> (<em>str</em>) &#x2013; SQL query to execute against the Microsoft SQL Server
database. (templated)</li>
<li><strong>hive_table</strong> (<em>str</em>) &#x2013; target Hive table, use dot notation to target a specific
database. (templated)</li>
<li><strong>create</strong> (<em>bool</em>) &#x2013; whether to create the table if it doesn&#x2019;t exist</li>
<li><strong>recreate</strong> (<em>bool</em>) &#x2013; whether to drop and recreate the table at every execution</li>
<li><strong>partition</strong> (<em>dict</em>) &#x2013; target partition as a dict of partition columns and
values. (templated)</li>
<li><strong>delimiter</strong> (<em>str</em>) &#x2013; field delimiter in the file</li>
<li><strong>mssql_conn_id</strong> (<em>str</em>) &#x2013; source Microsoft SQL Server connection</li>
<li><strong>hive_conn_id</strong> (<em>str</em>) &#x2013; destination hive connection</li>
<li><strong>tblproperties</strong> (<em>dict</em>) &#x2013; TBLPROPERTIES of the hive table being created</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.operators.mysql_operator.MySqlOperator(sql, mysql_conn_id=&apos;mysql_default&apos;, parameters=None, autocommit=False, database=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Executes sql code in a specific MySQL database</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>mysql_conn_id</strong> (<em>string</em>) &#x2013; reference to a specific mysql database</li>
<li><strong>sql</strong> (<em>Can receive a str representing a sql statement</em><em>,
</em><em>a list of str</em><em> (</em><em>sql statements</em><em>)</em><em>, or </em><em>reference to a template file.
Template reference are recognized by str ending in &apos;.sql&apos;</em>) &#x2013; the sql code to be executed. (templated)</li>
<li><strong>database</strong> (<em>string</em>) &#x2013; name of database which overwrite defined one in connection</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.operators.mysql_to_hive.MySqlToHiveTransfer(sql, hive_table, create=True, recreate=False, partition=None, delimiter=u&apos;x01&apos;, mysql_conn_id=&apos;mysql_default&apos;, hive_cli_conn_id=&apos;hive_cli_default&apos;, tblproperties=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Moves data from MySql to Hive. The operator runs your query against
MySQL, stores the file locally before loading it into a Hive table.
If the <code class="docutils literal notranslate"><span class="pre">create</span></code> or <code class="docutils literal notranslate"><span class="pre">recreate</span></code> arguments are set to <code class="docutils literal notranslate"><span class="pre">True</span></code>,
a <code class="docutils literal notranslate"><span class="pre">CREATE</span> <span class="pre">TABLE</span></code> and <code class="docutils literal notranslate"><span class="pre">DROP</span> <span class="pre">TABLE</span></code> statements are generated.
Hive data types are inferred from the cursor&#x2019;s metadata. Note that the
table generated in Hive uses <code class="docutils literal notranslate"><span class="pre">STORED</span> <span class="pre">AS</span> <span class="pre">textfile</span></code>
which isn&#x2019;t the most efficient serialization format. If a
large amount of data is loaded and/or if the table gets
queried considerably, you may want to use this operator only to
stage the data into a temporary table before loading it into its
final destination using a <code class="docutils literal notranslate"><span class="pre">HiveOperator</span></code>.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>sql</strong> (<em>str</em>) &#x2013; SQL query to execute against the MySQL database. (templated)</li>
<li><strong>hive_table</strong> (<em>str</em>) &#x2013; target Hive table, use dot notation to target a
specific database. (templated)</li>
<li><strong>create</strong> (<em>bool</em>) &#x2013; whether to create the table if it doesn&#x2019;t exist</li>
<li><strong>recreate</strong> (<em>bool</em>) &#x2013; whether to drop and recreate the table at every
execution</li>
<li><strong>partition</strong> (<em>dict</em>) &#x2013; target partition as a dict of partition columns
and values. (templated)</li>
<li><strong>delimiter</strong> (<em>str</em>) &#x2013; field delimiter in the file</li>
<li><strong>mysql_conn_id</strong> (<em>str</em>) &#x2013; source mysql connection</li>
<li><strong>hive_conn_id</strong> (<em>str</em>) &#x2013; destination hive connection</li>
<li><strong>tblproperties</strong> (<em>dict</em>) &#x2013; TBLPROPERTIES of the hive table being created</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.operators.oracle_operator.OracleOperator(sql, oracle_conn_id=&apos;oracle_default&apos;, parameters=None, autocommit=False, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Executes sql code in a specific Oracle database
:param oracle_conn_id: reference to a specific Oracle database
:type oracle_conn_id: string
:param sql: the sql code to be executed. (templated)
:type sql: Can receive a str representing a sql statement,</p>
<blockquote>
<div>a list of str (sql statements), or reference to a template file.
Template reference are recognized by str ending in &#x2018;.sql&#x2019;</div>
</blockquote>




<pre>
class airflow.operators.pig_operator.PigOperator(pig, pig_cli_conn_id=&apos;pig_cli_default&apos;, pigparams_jinja_translate=False, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Executes pig script.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>pig</strong> (<em>string</em>) &#x2013; the pig latin script to be executed. (templated)</li>
<li><strong>pig_cli_conn_id</strong> (<em>string</em>) &#x2013; reference to the Hive database</li>
<li><strong>pigparams_jinja_translate</strong> (<em>boolean</em>) &#x2013; when True, pig params-type templating
${var} gets translated into jinja-type templating {{ var }}. Note that
you may want to use this along with the
<code class="docutils literal notranslate"><span class="pre">DAG(user_defined_macros=myargs)</span></code> parameter. View the DAG
object documentation for more details.</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.operators.postgres_operator.PostgresOperator(sql, postgres_conn_id=&apos;postgres_default&apos;, autocommit=False, parameters=None, database=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Executes sql code in a specific Postgres database</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>postgres_conn_id</strong> (<em>string</em>) &#x2013; reference to a specific postgres database</li>
<li><strong>sql</strong> (<em>Can receive a str representing a sql statement</em><em>,
</em><em>a list of str</em><em> (</em><em>sql statements</em><em>)</em><em>, or </em><em>reference to a template file.
Template reference are recognized by str ending in &apos;.sql&apos;</em>) &#x2013; the sql code to be executed. (templated)</li>
<li><strong>database</strong> (<em>string</em>) &#x2013; name of database which overwrite defined one in connection</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.operators.presto_check_operator.PrestoCheckOperator(sql, presto_conn_id=&apos;presto_default&apos;, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.operators.check_operator.CheckOperator" title="airflow.operators.check_operator.CheckOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.operators.check_operator.CheckOperator</span></code></a></p>
<p>Performs checks against Presto. The <code class="docutils literal notranslate"><span class="pre">PrestoCheckOperator</span></code> expects
a sql query that will return a single row. Each value on that
first row is evaluated using python <code class="docutils literal notranslate"><span class="pre">bool</span></code> casting. If any of the
values return <code class="docutils literal notranslate"><span class="pre">False</span></code> the check is failed and errors out.</p>
<p>Note that Python bool casting evals the following as <code class="docutils literal notranslate"><span class="pre">False</span></code>:</p>
<ul class="simple">
<li><code class="docutils literal notranslate"><span class="pre">False</span></code></li>
<li><code class="docutils literal notranslate"><span class="pre">0</span></code></li>
<li>Empty string (<code class="docutils literal notranslate"><span class="pre">&quot;&quot;</span></code>)</li>
<li>Empty list (<code class="docutils literal notranslate"><span class="pre">[]</span></code>)</li>
<li>Empty dictionary or set (<code class="docutils literal notranslate"><span class="pre">{}</span></code>)</li>
</ul>
<p>Given a query like <code class="docutils literal notranslate"><span class="pre">SELECT</span> <span class="pre">COUNT(*)</span> <span class="pre">FROM</span> <span class="pre">foo</span></code>, it will fail only if
the count <code class="docutils literal notranslate"><span class="pre">==</span> <span class="pre">0</span></code>. You can craft much more complex query that could,
for instance, check that the table has the same number of rows as
the source table upstream, or that the count of today&#x2019;s partition is
greater than yesterday&#x2019;s partition, or that a set of metrics are less
than 3 standard deviation for the 7 day average.</p>
<p>This operator can be used as a data quality check in your pipeline, and
depending on where you put it in your DAG, you have the choice to
stop the critical path, preventing from
publishing dubious data, or on the side and receive email alterts
without stopping the progress of the DAG.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>sql</strong> (<em>string</em>) &#x2013; the sql to be executed</li>
<li><strong>presto_conn_id</strong> (<em>string</em>) &#x2013; reference to the Presto database</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.operators.presto_check_operator.PrestoIntervalCheckOperator(table, metrics_thresholds, date_filter_column=&apos;ds&apos;, days_back=-7, presto_conn_id=&apos;presto_default&apos;, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.operators.check_operator.IntervalCheckOperator" title="airflow.operators.check_operator.IntervalCheckOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.operators.check_operator.IntervalCheckOperator</span></code></a></p>
<p>Checks that the values of metrics given as SQL expressions are within
a certain tolerance of the ones from days_back before.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>table</strong> (<em>str</em>) &#x2013; the table name</li>
<li><strong>days_back</strong> (<em>int</em>) &#x2013; number of days between ds and the ds we want to check
against. Defaults to 7 days</li>
<li><strong>metrics_threshold</strong> (<em>dict</em>) &#x2013; a dictionary of ratios indexed by metrics</li>
<li><strong>presto_conn_id</strong> (<em>string</em>) &#x2013; reference to the Presto database</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.operators.presto_to_mysql.PrestoToMySqlTransfer(sql, mysql_table, presto_conn_id=&apos;presto_default&apos;, mysql_conn_id=&apos;mysql_default&apos;, mysql_preoperator=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Moves data from Presto to MySQL, note that for now the data is loaded
into memory before being pushed to MySQL, so this operator should
be used for smallish amount of data.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>sql</strong> (<em>str</em>) &#x2013; SQL query to execute against Presto. (templated)</li>
<li><strong>mysql_table</strong> (<em>str</em>) &#x2013; target MySQL table, use dot notation to target a
specific database. (templated)</li>
<li><strong>mysql_conn_id</strong> (<em>str</em>) &#x2013; source mysql connection</li>
<li><strong>presto_conn_id</strong> (<em>str</em>) &#x2013; source presto connection</li>
<li><strong>mysql_preoperator</strong> (<em>str</em>) &#x2013; sql statement to run against mysql prior to
import, typically use to truncate of delete in place
of the data coming in, allowing the task to be idempotent (running
the task twice won&#x2019;t double load data). (templated)</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.operators.presto_check_operator.PrestoValueCheckOperator(sql, pass_value, tolerance=None, presto_conn_id=&apos;presto_default&apos;, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.operators.check_operator.ValueCheckOperator" title="airflow.operators.check_operator.ValueCheckOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.operators.check_operator.ValueCheckOperator</span></code></a></p>
<p>Performs a simple value check using sql code.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>sql</strong> (<em>string</em>) &#x2013; the sql to be executed</li>
<li><strong>presto_conn_id</strong> (<em>string</em>) &#x2013; reference to the Presto database</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.operators.python_operator.PythonOperator(python_callable, op_args=None, op_kwargs=None, provide_context=False, templates_dict=None, templates_exts=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Executes a Python callable</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>python_callable</strong> (<em>python callable</em>) &#x2013; A reference to an object that is callable</li>
<li><strong>op_kwargs</strong> (<em>dict</em>) &#x2013; a dictionary of keyword arguments that will get unpacked
in your function</li>
<li><strong>op_args</strong> (<em>list</em>) &#x2013; a list of positional arguments that will get unpacked when
calling your callable</li>
<li><strong>provide_context</strong> (<em>bool</em>) &#x2013; if set to true, Airflow will pass a set of
keyword arguments that can be used in your function. This set of
kwargs correspond exactly to what you can use in your jinja
templates. For this to work, you need to define <cite>**kwargs</cite> in your
function header.</li>
<li><strong>templates_dict</strong> (<em>dict of str</em>) &#x2013; a dictionary where the values are templates that
will get templated by the Airflow engine sometime between
<code class="docutils literal notranslate"><span class="pre">__init__</span></code> and <code class="docutils literal notranslate"><span class="pre">execute</span></code> takes place and are made available
in your callable&#x2019;s context after the template has been applied. (templated)</li>
<li><strong>templates_exts</strong> (<em>list</em><em>(</em><em>str</em><em>)</em>) &#x2013; a list of file extensions to resolve while
processing templated fields, for examples <code class="docutils literal notranslate"><span class="pre">[&apos;.sql&apos;,</span> <span class="pre">&apos;.hql&apos;]</span></code></li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.operators.python_operator.PythonVirtualenvOperator(python_callable, requirements=None, python_version=None, use_dill=False, system_site_packages=True, op_args=None, op_kwargs=None, string_args=None, templates_dict=None, templates_exts=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.operators.python_operator.PythonOperator" title="airflow.operators.python_operator.PythonOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.operators.python_operator.PythonOperator</span></code></a></p>
<p>Allows one to run a function in a virtualenv that is created and destroyed
automatically (with certain caveats).</p>
<p>The function must be defined using def, and not be
part of a class. All imports must happen inside the function
and no variables outside of the scope may be referenced. A global scope
variable named virtualenv_string_args will be available (populated by
string_args). In addition, one can pass stuff through op_args and op_kwargs, and one
can use a return value.</p>
<p>Note that if your virtualenv runs in a different Python major version than Airflow,
you cannot use return values, op_args, or op_kwargs. You can use string_args though.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>python_callable</strong> (<em>function</em>) &#x2013; A python function with no references to outside variables,
defined with def, which will be run in a virtualenv</li>
<li><strong>requirements</strong> (<em>list</em><em>(</em><em>str</em><em>)</em>) &#x2013; A list of requirements as specified in a pip install command</li>
<li><strong>python_version</strong> (<em>str</em>) &#x2013; The Python version to run the virtualenv with. Note that
both 2 and 2.7 are acceptable forms.</li>
<li><strong>use_dill</strong> (<em>bool</em>) &#x2013; Whether to use dill to serialize
the args and result (pickle is default). This allow more complex types
but requires you to include dill in your requirements.</li>
<li><strong>system_site_packages</strong> (<em>bool</em>) &#x2013; Whether to include
system_site_packages in your virtualenv.
See virtualenv documentation for more information.</li>
<li><strong>op_args</strong> &#x2013; A list of positional arguments to pass to python_callable.</li>
<li><strong>op_kwargs</strong> (<em>dict</em>) &#x2013; A dict of keyword arguments to pass to python_callable.</li>
<li><strong>string_args</strong> (<em>list</em><em>(</em><em>str</em><em>)</em>) &#x2013; Strings that are present in the global var virtualenv_string_args,
available to python_callable at runtime as a list(str). Note that args are split
by newline.</li>
<li><strong>templates_dict</strong> (<em>dict of str</em>) &#x2013; a dictionary where the values are templates that
will get templated by the Airflow engine sometime between
<code class="docutils literal notranslate"><span class="pre">__init__</span></code> and <code class="docutils literal notranslate"><span class="pre">execute</span></code> takes place and are made available
in your callable&#x2019;s context after the template has been applied</li>
<li><strong>templates_exts</strong> (<em>list</em><em>(</em><em>str</em><em>)</em>) &#x2013; a list of file extensions to resolve while
processing templated fields, for examples <code class="docutils literal notranslate"><span class="pre">[&apos;.sql&apos;,</span> <span class="pre">&apos;.hql&apos;]</span></code></li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.operators.s3_file_transform_operator.S3FileTransformOperator(source_s3_key, dest_s3_key, transform_script=None, select_expression=None, source_aws_conn_id=&apos;aws_default&apos;, dest_aws_conn_id=&apos;aws_default&apos;, replace=False, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Copies data from a source S3 location to a temporary location on the
local filesystem. Runs a transformation on this file as specified by
the transformation script and uploads the output to a destination S3
location.</p>
<p>The locations of the source and the destination files in the local
filesystem is provided as an first and second arguments to the
transformation script. The transformation script is expected to read the
data from source, transform it and write the output to the local
destination file. The operator then takes over control and uploads the
local destination file to S3.</p>
<p>S3 Select is also available to filter the source contents. Users can
omit the transformation script if S3 Select expression is specified.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>source_s3_key</strong> (<em>str</em>) &#x2013; The key to be retrieved from S3. (templated)</li>
<li><strong>source_aws_conn_id</strong> (<em>str</em>) &#x2013; source s3 connection</li>
<li><strong>dest_s3_key</strong> (<em>str</em>) &#x2013; The key to be written from S3. (templated)</li>
<li><strong>dest_aws_conn_id</strong> (<em>str</em>) &#x2013; destination s3 connection</li>
<li><strong>replace</strong> (<em>bool</em>) &#x2013; Replace dest S3 key if it already exists</li>
<li><strong>transform_script</strong> (<em>str</em>) &#x2013; location of the executable transformation script</li>
<li><strong>select_expression</strong> (<em>str</em>) &#x2013; S3 Select expression</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.operators.s3_to_hive_operator.S3ToHiveTransfer(s3_key, field_dict, hive_table, delimiter=&apos;, &apos;, create=True, recreate=False, partition=None, headers=False, check_headers=False, wildcard_match=False, aws_conn_id=&apos;aws_default&apos;, hive_cli_conn_id=&apos;hive_cli_default&apos;, input_compressed=False, tblproperties=None, select_expression=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Moves data from S3 to Hive. The operator downloads a file from S3,
stores the file locally before loading it into a Hive table.
If the <code class="docutils literal notranslate"><span class="pre">create</span></code> or <code class="docutils literal notranslate"><span class="pre">recreate</span></code> arguments are set to <code class="docutils literal notranslate"><span class="pre">True</span></code>,
a <code class="docutils literal notranslate"><span class="pre">CREATE</span> <span class="pre">TABLE</span></code> and <code class="docutils literal notranslate"><span class="pre">DROP</span> <span class="pre">TABLE</span></code> statements are generated.
Hive data types are inferred from the cursor&#x2019;s metadata from.</p>
<p>Note that the table generated in Hive uses <code class="docutils literal notranslate"><span class="pre">STORED</span> <span class="pre">AS</span> <span class="pre">textfile</span></code>
which isn&#x2019;t the most efficient serialization format. If a
large amount of data is loaded and/or if the tables gets
queried considerably, you may want to use this operator only to
stage the data into a temporary table before loading it into its
final destination using a <code class="docutils literal notranslate"><span class="pre">HiveOperator</span></code>.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>s3_key</strong> (<em>str</em>) &#x2013; The key to be retrieved from S3. (templated)</li>
<li><strong>field_dict</strong> (<em>dict</em>) &#x2013; A dictionary of the fields name in the file
as keys and their Hive types as values</li>
<li><strong>hive_table</strong> (<em>str</em>) &#x2013; target Hive table, use dot notation to target a
specific database. (templated)</li>
<li><strong>create</strong> (<em>bool</em>) &#x2013; whether to create the table if it doesn&#x2019;t exist</li>
<li><strong>recreate</strong> (<em>bool</em>) &#x2013; whether to drop and recreate the table at every
execution</li>
<li><strong>partition</strong> (<em>dict</em>) &#x2013; target partition as a dict of partition columns
and values. (templated)</li>
<li><strong>headers</strong> (<em>bool</em>) &#x2013; whether the file contains column names on the first
line</li>
<li><strong>check_headers</strong> (<em>bool</em>) &#x2013; whether the column names on the first line should be
checked against the keys of field_dict</li>
<li><strong>wildcard_match</strong> (<em>bool</em>) &#x2013; whether the s3_key should be interpreted as a Unix
wildcard pattern</li>
<li><strong>delimiter</strong> (<em>str</em>) &#x2013; field delimiter in the file</li>
<li><strong>aws_conn_id</strong> (<em>str</em>) &#x2013; source s3 connection</li>
<li><strong>hive_cli_conn_id</strong> (<em>str</em>) &#x2013; destination hive connection</li>
<li><strong>input_compressed</strong> (<em>bool</em>) &#x2013; Boolean to determine if file decompression is
required to process headers</li>
<li><strong>tblproperties</strong> (<em>dict</em>) &#x2013; TBLPROPERTIES of the hive table being created</li>
<li><strong>select_expression</strong> (<em>str</em>) &#x2013; S3 Select expression</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.operators.s3_to_redshift_operator.S3ToRedshiftTransfer(schema, table, s3_bucket, s3_key, redshift_conn_id=&apos;redshift_default&apos;, aws_conn_id=&apos;aws_default&apos;, copy_options=(), autocommit=False, parameters=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Executes an COPY command to load files from s3 to Redshift</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>schema</strong> (<em>string</em>) &#x2013; reference to a specific schema in redshift database</li>
<li><strong>table</strong> (<em>string</em>) &#x2013; reference to a specific table in redshift database</li>
<li><strong>s3_bucket</strong> (<em>string</em>) &#x2013; reference to a specific S3 bucket</li>
<li><strong>s3_key</strong> (<em>string</em>) &#x2013; reference to a specific S3 key</li>
<li><strong>redshift_conn_id</strong> (<em>string</em>) &#x2013; reference to a specific redshift database</li>
<li><strong>aws_conn_id</strong> (<em>string</em>) &#x2013; reference to a specific S3 connection</li>
<li><strong>copy_options</strong> (<em>list</em>) &#x2013; reference to a list of COPY options</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.operators.python_operator.ShortCircuitOperator(python_callable, op_args=None, op_kwargs=None, provide_context=False, templates_dict=None, templates_exts=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.operators.python_operator.PythonOperator" title="airflow.operators.python_operator.PythonOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.operators.python_operator.PythonOperator</span></code></a>, <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.SkipMixin</span></code></p>
<p>Allows a workflow to continue only if a condition is met. Otherwise, the
workflow &#x201C;short-circuits&#x201D; and downstream tasks are skipped.</p>
<p>The ShortCircuitOperator is derived from the PythonOperator. It evaluates a
condition and short-circuits the workflow if the condition is False. Any
downstream tasks are marked with a state of &#x201C;skipped&#x201D;. If the condition is
True, downstream tasks proceed as normal.</p>
<p>The condition is determined by the result of <cite>python_callable</cite>.</p>




<pre>
class airflow.operators.http_operator.SimpleHttpOperator(endpoint, method=&apos;POST&apos;, data=None, headers=None, response_check=None, extra_options=None, xcom_push=False, http_conn_id=&apos;http_default&apos;, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Calls an endpoint on an HTTP system to execute an action</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>http_conn_id</strong> (<em>string</em>) &#x2013; The connection to run the sensor against</li>
<li><strong>endpoint</strong> (<em>string</em>) &#x2013; The relative part of the full url. (templated)</li>
<li><strong>method</strong> (<em>string</em>) &#x2013; The HTTP method to use, default = &#x201C;POST&#x201D;</li>
<li><strong>data</strong> (<em>For POST/PUT</em><em>, </em><em>depends on the content-type parameter</em><em>,
</em><em>for GET a dictionary of key/value string pairs</em>) &#x2013; The data to pass. POST-data in POST/PUT and params
in the URL for a GET request. (templated)</li>
<li><strong>headers</strong> (<em>a dictionary of string key/value pairs</em>) &#x2013; The HTTP headers to be added to the GET request</li>
<li><strong>response_check</strong> (<em>A lambda</em><em> or </em><em>defined function.</em>) &#x2013; A check against the &#x2018;requests&#x2019; response object.
Returns True for &#x2018;pass&#x2019; and False otherwise.</li>
<li><strong>extra_options</strong> (<em>A dictionary of options</em><em>, </em><em>where key is string and value
depends on the option that&apos;s being modified.</em>) &#x2013; Extra options for the &#x2018;requests&#x2019; library, see the
&#x2018;requests&#x2019; documentation (options to modify timeout, ssl, etc.)</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.operators.slack_operator.SlackAPIOperator(slack_conn_id=None, token=None, method=None, api_params=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Base Slack Operator
The SlackAPIPostOperator is derived from this operator.
In the future additional Slack API Operators will be derived from this class as well</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>slack_conn_id</strong> (<em>string</em>) &#x2013; Slack connection ID which its password is Slack API token</li>
<li><strong>token</strong> (<em>string</em>) &#x2013; Slack API token (<a class="reference external" href="https://api.slack.com/web">https://api.slack.com/web</a>)</li>
<li><strong>method</strong> (<em>string</em>) &#x2013; The Slack API Method to Call (<a class="reference external" href="https://api.slack.com/methods">https://api.slack.com/methods</a>)</li>
<li><strong>api_params</strong> (<em>dict</em>) &#x2013; API Method call parameters (<a class="reference external" href="https://api.slack.com/methods">https://api.slack.com/methods</a>)</li>
</ul>
</td>
</tr>
</tbody>
</table>

<pre>
construct_api_call_params()</pre>
<p>Used by the execute function. Allows templating on the source fields
of the api_call_params dict before construction</p>
<p>Override in child classes.
Each SlackAPIOperator child class is responsible for
having a construct_api_call_params function
which sets self.api_call_params with a dict of
API call parameters (<a class="reference external" href="https://api.slack.com/methods">https://api.slack.com/methods</a>)</p>




<pre>
execute(**kwargs)</pre>
<p>SlackAPIOperator calls will not fail even if the call is not unsuccessful.
It should not prevent a DAG from completing in success</p>







<pre>
class airflow.operators.slack_operator.SlackAPIPostOperator(channel=&apos;#general&apos;, username=&apos;Airflow&apos;, text=&apos;No message has been set.nHere is a cat video insteadnhttps://www.youtube.com/watch?v=J---aiyznGQ&apos;, icon_url=&apos;https://raw.githubusercontent.com/airbnb/airflow/master/airflow/www/static/pin_100.png&apos;, attachments=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.operators.slack_operator.SlackAPIOperator" title="airflow.operators.slack_operator.SlackAPIOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.operators.slack_operator.SlackAPIOperator</span></code></a></p>
<p>Posts messages to a slack channel</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>channel</strong> (<em>string</em>) &#x2013; channel in which to post message on slack name (#general) or
ID (C12318391). (templated)</li>
<li><strong>username</strong> (<em>string</em>) &#x2013; Username that airflow will be posting to Slack as. (templated)</li>
<li><strong>text</strong> (<em>string</em>) &#x2013; message to send to slack. (templated)</li>
<li><strong>icon_url</strong> (<em>string</em>) &#x2013; url to icon used for this message</li>
<li><strong>attachments</strong> (<em>array of hashes</em>) &#x2013; extra formatting details. (templated)
- see <a class="reference external" href="https://api.slack.com/docs/attachments">https://api.slack.com/docs/attachments</a>.</li>
</ul>
</td>
</tr>
</tbody>
</table>

<pre>
construct_api_call_params()</pre>
<p>Used by the execute function. Allows templating on the source fields
of the api_call_params dict before construction</p>
<p>Override in child classes.
Each SlackAPIOperator child class is responsible for
having a construct_api_call_params function
which sets self.api_call_params with a dict of
API call parameters (<a class="reference external" href="https://api.slack.com/methods">https://api.slack.com/methods</a>)</p>







<pre>
class airflow.operators.sqlite_operator.SqliteOperator(sql, sqlite_conn_id=&apos;sqlite_default&apos;, parameters=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Executes sql code in a specific Sqlite database</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>sqlite_conn_id</strong> (<em>string</em>) &#x2013; reference to a specific sqlite database</li>
<li><strong>sql</strong> (<em>string</em><em> or </em><em>string pointing to a template file. File must have
a &apos;.sql&apos; extensions.</em>) &#x2013; the sql code to be executed. (templated)</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.operators.subdag_operator.SubDagOperator(**kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>




<pre>
class airflow.operators.dagrun_operator.TriggerDagRunOperator(trigger_dag_id, python_callable=None, execution_date=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Triggers a DAG run for a specified <code class="docutils literal notranslate"><span class="pre">dag_id</span></code></p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>trigger_dag_id</strong> (<em>str</em>) &#x2013; the dag_id to trigger</li>
<li><strong>python_callable</strong> (<em>python callable</em>) &#x2013; a reference to a python function that will be
called while passing it the <code class="docutils literal notranslate"><span class="pre">context</span></code> object and a placeholder
object <code class="docutils literal notranslate"><span class="pre">obj</span></code> for your callable to fill and return if you want
a DagRun created. This <code class="docutils literal notranslate"><span class="pre">obj</span></code> object contains a <code class="docutils literal notranslate"><span class="pre">run_id</span></code> and
<code class="docutils literal notranslate"><span class="pre">payload</span></code> attribute that you can modify in your function.
The <code class="docutils literal notranslate"><span class="pre">run_id</span></code> should be a unique identifier for that DAG run, and
the payload has to be a picklable object that will be made available
to your tasks while executing that DAG run. Your function header
should look like <code class="docutils literal notranslate"><span class="pre">def</span> <span class="pre">foo(context,</span> <span class="pre">dag_run_obj):</span></code></li>
<li><strong>execution_date</strong> (<em>datetime.datetime</em>) &#x2013; Execution date for the dag</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.operators.check_operator.ValueCheckOperator(sql, pass_value, tolerance=None, conn_id=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Performs a simple value check using sql code.</p>
<p>Note that this is an abstract class and get_db_hook
needs to be defined. Whereas a get_db_hook is hook that gets a
single record from an external source.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><strong>sql</strong> (<em>string</em>) &#x2013; the sql to be executed. (templated)</td>
</tr>
</tbody>
</table>




<pre>
class airflow.operators.redshift_to_s3_operator.RedshiftToS3Transfer(schema, table, s3_bucket, s3_key, redshift_conn_id=&apos;redshift_default&apos;, aws_conn_id=&apos;aws_default&apos;, unload_options=(), autocommit=False, parameters=None, include_header=False, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Executes an UNLOAD command to s3 as a CSV with headers</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>schema</strong> (<em>string</em>) &#x2013; reference to a specific schema in redshift database</li>
<li><strong>table</strong> (<em>string</em>) &#x2013; reference to a specific table in redshift database</li>
<li><strong>s3_bucket</strong> (<em>string</em>) &#x2013; reference to a specific S3 bucket</li>
<li><strong>s3_key</strong> (<em>string</em>) &#x2013; reference to a specific S3 key</li>
<li><strong>redshift_conn_id</strong> (<em>string</em>) &#x2013; reference to a specific redshift database</li>
<li><strong>aws_conn_id</strong> (<em>string</em>) &#x2013; reference to a specific S3 connection</li>
<li><strong>unload_options</strong> (<em>list</em>) &#x2013; reference to a list of UNLOAD options</li>
</ul>
</td>
</tr>
</tbody>
</table>



</div>
<div class="section" id="sensors">
<h4 class="sigil_not_in_toc">Sensors</h4>

<pre>
class airflow.sensors.external_task_sensor.ExternalTaskSensor(external_dag_id, external_task_id, allowed_states=None, execution_delta=None, execution_date_fn=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.sensors.base_sensor_operator.BaseSensorOperator" title="airflow.sensors.base_sensor_operator.BaseSensorOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.sensors.base_sensor_operator.BaseSensorOperator</span></code></a></p>
<p>Waits for a task to complete in a different DAG</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>external_dag_id</strong> (<em>string</em>) &#x2013; The dag_id that contains the task you want to
wait for</li>
<li><strong>external_task_id</strong> (<em>string</em>) &#x2013; The task_id that contains the task you want to
wait for</li>
<li><strong>allowed_states</strong> (<em>list</em>) &#x2013; list of allowed states, default is <code class="docutils literal notranslate"><span class="pre">[&apos;success&apos;]</span></code></li>
<li><strong>execution_delta</strong> (<em>datetime.timedelta</em>) &#x2013; time difference with the previous execution to
look at, the default is the same execution_date as the current task.
For yesterday, use [positive!] datetime.timedelta(days=1). Either
execution_delta or execution_date_fn can be passed to
ExternalTaskSensor, but not both.</li>
<li><strong>execution_date_fn</strong> (<em>callable</em>) &#x2013; function that receives the current execution date
and returns the desired execution dates to query. Either execution_delta
or execution_date_fn can be passed to ExternalTaskSensor, but not both.</li>
</ul>
</td>
</tr>
</tbody>
</table>

<pre>
poke(**kwargs)</pre>
<p>Function that the sensors defined while deriving this class should
override.</p>







<pre>
class airflow.sensors.hdfs_sensor.HdfsSensor(filepath, hdfs_conn_id=&apos;hdfs_default&apos;, ignored_ext=[&apos;_COPYING_&apos;], ignore_copying=True, file_size=None, hook=&lt;class &apos;airflow.hooks.hdfs_hook.HDFSHook&apos;&gt;, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.sensors.base_sensor_operator.BaseSensorOperator" title="airflow.sensors.base_sensor_operator.BaseSensorOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.sensors.base_sensor_operator.BaseSensorOperator</span></code></a></p>
<p>Waits for a file or folder to land in HDFS</p>

<pre>
static filter_for_filesize(result, size=None)</pre>
<p>Will test the filepath result and test if its size is at least self.filesize</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first simple">
<li><strong>result</strong> &#x2013; a list of dicts returned by Snakebite ls</li>
<li><strong>size</strong> &#x2013; the file size in MB a file should be at least to trigger True</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th>
<td class="field-body"><p class="first last">(bool) depending on the matching criteria</p>
</td>
</tr>
</tbody>
</table>




<pre>
static filter_for_ignored_ext(result, ignored_ext, ignore_copying)</pre>
<p>Will filter if instructed to do so the result to remove matching criteria</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first simple">
<li><strong>result</strong> &#x2013; (list) of dicts returned by Snakebite ls</li>
<li><strong>ignored_ext</strong> &#x2013; (list) of ignored extensions</li>
<li><strong>ignore_copying</strong> &#x2013; (bool) shall we ignore ?</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th>
<td class="field-body"><p class="first last">(list) of dicts which were not removed</p>
</td>
</tr>
</tbody>
</table>




<pre>
poke(context)</pre>
<p>Function that the sensors defined while deriving this class should
override.</p>







<pre>
class airflow.sensors.hive_partition_sensor.HivePartitionSensor(table, partition=&quot;ds=&apos;{{ ds }}&apos;&quot;, metastore_conn_id=&apos;metastore_default&apos;, schema=&apos;default&apos;, poke_interval=180, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.sensors.base_sensor_operator.BaseSensorOperator" title="airflow.sensors.base_sensor_operator.BaseSensorOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.sensors.base_sensor_operator.BaseSensorOperator</span></code></a></p>
<p>Waits for a partition to show up in Hive.</p>
<p>Note: Because <code class="docutils literal notranslate"><span class="pre">partition</span></code> supports general logical operators, it
can be inefficient. Consider using NamedHivePartitionSensor instead if
you don&#x2019;t need the full flexibility of HivePartitionSensor.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>table</strong> (<em>string</em>) &#x2013; The name of the table to wait for, supports the dot
notation (my_database.my_table)</li>
<li><strong>partition</strong> (<em>string</em>) &#x2013; The partition clause to wait for. This is passed as
is to the metastore Thrift client <code class="docutils literal notranslate"><span class="pre">get_partitions_by_filter</span></code> method,
and apparently supports SQL like notation as in <code class="docutils literal notranslate"><span class="pre">ds=&apos;2015-01-01&apos;</span>
<span class="pre">AND</span> <span class="pre">type=&apos;value&apos;</span></code> and comparison operators as in <code class="docutils literal notranslate"><span class="pre">&quot;ds&gt;=2015-01-01&quot;</span></code></li>
<li><strong>metastore_conn_id</strong> (<em>str</em>) &#x2013; reference to the metastore thrift service
connection id</li>
</ul>
</td>
</tr>
</tbody>
</table>

<pre>
poke(context)</pre>
<p>Function that the sensors defined while deriving this class should
override.</p>







<pre>
class airflow.sensors.http_sensor.HttpSensor(endpoint, http_conn_id=&apos;http_default&apos;, method=&apos;GET&apos;, request_params=None, headers=None, response_check=None, extra_options=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.sensors.base_sensor_operator.BaseSensorOperator" title="airflow.sensors.base_sensor_operator.BaseSensorOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.sensors.base_sensor_operator.BaseSensorOperator</span></code></a></p>

<pre>Executes a HTTP get statement and returns False on failure:</pre>
404 not found or response_check function returned False

<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>http_conn_id</strong> (<em>string</em>) &#x2013; The connection to run the sensor against</li>
<li><strong>method</strong> (<em>string</em>) &#x2013; The HTTP request method to use</li>
<li><strong>endpoint</strong> (<em>string</em>) &#x2013; The relative part of the full url</li>
<li><strong>request_params</strong> (<em>a dictionary of string key/value pairs</em>) &#x2013; The parameters to be added to the GET url</li>
<li><strong>headers</strong> (<em>a dictionary of string key/value pairs</em>) &#x2013; The HTTP headers to be added to the GET request</li>
<li><strong>response_check</strong> (<em>A lambda</em><em> or </em><em>defined function.</em>) &#x2013; A check against the &#x2018;requests&#x2019; response object.
Returns True for &#x2018;pass&#x2019; and False otherwise.</li>
<li><strong>extra_options</strong> (<em>A dictionary of options</em><em>, </em><em>where key is string and value
depends on the option that&apos;s being modified.</em>) &#x2013; Extra options for the &#x2018;requests&#x2019; library, see the
&#x2018;requests&#x2019; documentation (options to modify timeout, ssl, etc.)</li>
</ul>
</td>
</tr>
</tbody>
</table>

<pre>
poke(context)</pre>
<p>Function that the sensors defined while deriving this class should
override.</p>







<pre>
class airflow.sensors.metastore_partition_sensor.MetastorePartitionSensor(table, partition_name, schema=&apos;default&apos;, mysql_conn_id=&apos;metastore_mysql&apos;, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.sensors.sql_sensor.SqlSensor" title="airflow.sensors.sql_sensor.SqlSensor"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.sensors.sql_sensor.SqlSensor</span></code></a></p>
<p>An alternative to the HivePartitionSensor that talk directly to the
MySQL db. This was created as a result of observing sub optimal
queries generated by the Metastore thrift service when hitting
subpartitioned tables. The Thrift service&#x2019;s queries were written in a
way that wouldn&#x2019;t leverage the indexes.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>schema</strong> (<em>str</em>) &#x2013; the schema</li>
<li><strong>table</strong> (<em>str</em>) &#x2013; the table</li>
<li><strong>partition_name</strong> (<em>str</em>) &#x2013; the partition name, as defined in the PARTITIONS
table of the Metastore. Order of the fields does matter.
Examples: <code class="docutils literal notranslate"><span class="pre">ds=2016-01-01</span></code> or
<code class="docutils literal notranslate"><span class="pre">ds=2016-01-01/sub=foo</span></code> for a sub partitioned table</li>
<li><strong>mysql_conn_id</strong> (<em>str</em>) &#x2013; a reference to the MySQL conn_id for the metastore</li>
</ul>
</td>
</tr>
</tbody>
</table>

<pre>
poke(context)</pre>
<p>Function that the sensors defined while deriving this class should
override.</p>







<pre>
class airflow.sensors.named_hive_partition_sensor.NamedHivePartitionSensor(partition_names, metastore_conn_id=&apos;metastore_default&apos;, poke_interval=180, hook=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.sensors.base_sensor_operator.BaseSensorOperator" title="airflow.sensors.base_sensor_operator.BaseSensorOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.sensors.base_sensor_operator.BaseSensorOperator</span></code></a></p>
<p>Waits for a set of partitions to show up in Hive.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>partition_names</strong> (<em>list of strings</em>) &#x2013; List of fully qualified names of the
partitions to wait for. A fully qualified name is of the
form <code class="docutils literal notranslate"><span class="pre">schema.table/pk1=pv1/pk2=pv2</span></code>, for example,
default.users/ds=2016-01-01. This is passed as is to the metastore
Thrift client <code class="docutils literal notranslate"><span class="pre">get_partitions_by_name</span></code> method. Note that
you cannot use logical or comparison operators as in
HivePartitionSensor.</li>
<li><strong>metastore_conn_id</strong> (<em>str</em>) &#x2013; reference to the metastore thrift service
connection id</li>
</ul>
</td>
</tr>
</tbody>
</table>

<pre>
poke(context)</pre>
<p>Function that the sensors defined while deriving this class should
override.</p>







<pre>
class airflow.sensors.s3_key_sensor.S3KeySensor(bucket_key, bucket_name=None, wildcard_match=False, aws_conn_id=&apos;aws_default&apos;, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.sensors.base_sensor_operator.BaseSensorOperator" title="airflow.sensors.base_sensor_operator.BaseSensorOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.sensors.base_sensor_operator.BaseSensorOperator</span></code></a></p>
<p>Waits for a key (a file-like instance on S3) to be present in a S3 bucket.
S3 being a key/value it does not support folders. The path is just a key
a resource.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>bucket_key</strong> (<em>str</em>) &#x2013; The key being waited on. Supports full s3:// style url
or relative path from root level.</li>
<li><strong>bucket_name</strong> (<em>str</em>) &#x2013; Name of the S3 bucket</li>
<li><strong>wildcard_match</strong> (<em>bool</em>) &#x2013; whether the bucket_key should be interpreted as a
Unix wildcard pattern</li>
<li><strong>aws_conn_id</strong> (<em>str</em>) &#x2013; a reference to the s3 connection</li>
</ul>
</td>
</tr>
</tbody>
</table>

<pre>
poke(context)</pre>
<p>Function that the sensors defined while deriving this class should
override.</p>







<pre>
class airflow.sensors.s3_prefix_sensor.S3PrefixSensor(bucket_name, prefix, delimiter=&apos;/&apos;, aws_conn_id=&apos;aws_default&apos;, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.sensors.base_sensor_operator.BaseSensorOperator" title="airflow.sensors.base_sensor_operator.BaseSensorOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.sensors.base_sensor_operator.BaseSensorOperator</span></code></a></p>
<p>Waits for a prefix to exist. A prefix is the first part of a key,
thus enabling checking of constructs similar to glob airfl* or
SQL LIKE &#x2018;airfl%&#x2019;. There is the possibility to precise a delimiter to
indicate the hierarchy or keys, meaning that the match will stop at that
delimiter. Current code accepts sane delimiters, i.e. characters that
are NOT special characters in the Python regex engine.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>bucket_name</strong> (<em>str</em>) &#x2013; Name of the S3 bucket</li>
<li><strong>prefix</strong> (<em>str</em>) &#x2013; The prefix being waited on. Relative path from bucket root level.</li>
<li><strong>delimiter</strong> (<em>str</em>) &#x2013; The delimiter intended to show hierarchy.
Defaults to &#x2018;/&#x2019;.</li>
</ul>
</td>
</tr>
</tbody>
</table>

<pre>
poke(context)</pre>
<p>Function that the sensors defined while deriving this class should
override.</p>







<pre>
class airflow.sensors.sql_sensor.SqlSensor(conn_id, sql, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.sensors.base_sensor_operator.BaseSensorOperator" title="airflow.sensors.base_sensor_operator.BaseSensorOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.sensors.base_sensor_operator.BaseSensorOperator</span></code></a></p>
<p>Runs a sql statement until a criteria is met. It will keep trying while
sql returns no row, or if the first cell in (0, &#x2018;0&#x2019;, &#x2018;&#x2019;).</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>conn_id</strong> (<em>string</em>) &#x2013; The connection to run the sensor against</li>
<li><strong>sql</strong> &#x2013; The sql to run. To pass, it needs to return at least one cell
that contains a non-zero / empty string value.</li>
</ul>
</td>
</tr>
</tbody>
</table>

<pre>
poke(context)</pre>
<p>Function that the sensors defined while deriving this class should
override.</p>







<pre>
class airflow.sensors.time_sensor.TimeSensor(target_time, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.sensors.base_sensor_operator.BaseSensorOperator" title="airflow.sensors.base_sensor_operator.BaseSensorOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.sensors.base_sensor_operator.BaseSensorOperator</span></code></a></p>
<p>Waits until the specified time of the day.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><strong>target_time</strong> (<em>datetime.time</em>) &#x2013; time after which the job succeeds</td>
</tr>
</tbody>
</table>

<pre>
poke(context)</pre>
<p>Function that the sensors defined while deriving this class should
override.</p>







<pre>
class airflow.sensors.time_delta_sensor.TimeDeltaSensor(delta, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.sensors.base_sensor_operator.BaseSensorOperator" title="airflow.sensors.base_sensor_operator.BaseSensorOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.sensors.base_sensor_operator.BaseSensorOperator</span></code></a></p>
<p>Waits for a timedelta after the task&#x2019;s execution_date + schedule_interval.
In Airflow, the daily task stamped with <code class="docutils literal notranslate"><span class="pre">execution_date</span></code>
2016-01-01 can only start running on 2016-01-02. The timedelta here
represents the time after the execution period has closed.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><strong>delta</strong> (<em>datetime.timedelta</em>) &#x2013; time length to wait after execution_date before succeeding</td>
</tr>
</tbody>
</table>

<pre>
poke(context)</pre>
<p>Function that the sensors defined while deriving this class should
override.</p>







<pre>
class airflow.sensors.web_hdfs_sensor.WebHdfsSensor(filepath, webhdfs_conn_id=&apos;webhdfs_default&apos;, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.sensors.base_sensor_operator.BaseSensorOperator" title="airflow.sensors.base_sensor_operator.BaseSensorOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.sensors.base_sensor_operator.BaseSensorOperator</span></code></a></p>
<p>Waits for a file or folder to land in HDFS</p>

<pre>
poke(context)</pre>
<p>Function that the sensors defined while deriving this class should
override.</p>






</div>
</div>
<div class="section" id="community-contributed-operators">
<h3 class="sigil_not_in_toc">Community-contributed Operators</h3>
<div class="section" id="id4">
<h4 class="sigil_not_in_toc">Operators</h4>

<pre>
class airflow.contrib.operators.awsbatch_operator.AWSBatchOperator(job_name, job_definition, job_queue, overrides, max_retries=4200, aws_conn_id=None, region_name=None, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Execute a job on AWS Batch Service</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first simple">
<li><strong>job_name</strong> (<em>str</em>) &#x2013; the name for the job that will run on AWS Batch</li>
<li><strong>job_definition</strong> (<em>str</em>) &#x2013; the job definition name on AWS Batch</li>
<li><strong>job_queue</strong> (<em>str</em>) &#x2013; the queue name on AWS Batch</li>
<li><strong>max_retries</strong> (<em>int</em>) &#x2013; exponential backoff retries while waiter is not merged, 4200 = 48 hours</li>
<li><strong>aws_conn_id</strong> (<em>str</em>) &#x2013; connection id of AWS credentials / region name. If None,
credential boto3 strategy will be used
(<a class="reference external" href="http://boto3.readthedocs.io/en/latest/guide/configuration.html">http://boto3.readthedocs.io/en/latest/guide/configuration.html</a>).</li>
<li><strong>region_name</strong> &#x2013; region name to use in AWS Hook.
Override the region_name in connection (if provided)</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Param:</th>
<td class="field-body"><p class="first">overrides: the same parameter that boto3 will receive on
containerOverrides (templated):
<a class="reference external" href="http://boto3.readthedocs.io/en/latest/reference/services/batch.html#submit_job">http://boto3.readthedocs.io/en/latest/reference/services/batch.html#submit_job</a></p>
</td>
</tr>
<tr class="field-odd field"><th class="field-name">Type:</th>
<td class="field-body"><p class="first last">overrides: dict</p>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.contrib.operators.bigquery_check_operator.BigQueryCheckOperator(sql, bigquery_conn_id=&apos;bigquery_default&apos;, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.operators.check_operator.CheckOperator" title="airflow.operators.check_operator.CheckOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.operators.check_operator.CheckOperator</span></code></a></p>
<p>Performs checks against BigQuery. The <code class="docutils literal notranslate"><span class="pre">BigQueryCheckOperator</span></code> expects
a sql query that will return a single row. Each value on that
first row is evaluated using python <code class="docutils literal notranslate"><span class="pre">bool</span></code> casting. If any of the
values return <code class="docutils literal notranslate"><span class="pre">False</span></code> the check is failed and errors out.</p>
<p>Note that Python bool casting evals the following as <code class="docutils literal notranslate"><span class="pre">False</span></code>:</p>
<ul class="simple">
<li><code class="docutils literal notranslate"><span class="pre">False</span></code></li>
<li><code class="docutils literal notranslate"><span class="pre">0</span></code></li>
<li>Empty string (<code class="docutils literal notranslate"><span class="pre">&quot;&quot;</span></code>)</li>
<li>Empty list (<code class="docutils literal notranslate"><span class="pre">[]</span></code>)</li>
<li>Empty dictionary or set (<code class="docutils literal notranslate"><span class="pre">{}</span></code>)</li>
</ul>
<p>Given a query like <code class="docutils literal notranslate"><span class="pre">SELECT</span> <span class="pre">COUNT(*)</span> <span class="pre">FROM</span> <span class="pre">foo</span></code>, it will fail only if
the count <code class="docutils literal notranslate"><span class="pre">==</span> <span class="pre">0</span></code>. You can craft much more complex query that could,
for instance, check that the table has the same number of rows as
the source table upstream, or that the count of today&#x2019;s partition is
greater than yesterday&#x2019;s partition, or that a set of metrics are less
than 3 standard deviation for the 7 day average.</p>
<p>This operator can be used as a data quality check in your pipeline, and
depending on where you put it in your DAG, you have the choice to
stop the critical path, preventing from
publishing dubious data, or on the side and receive email alterts
without stopping the progress of the DAG.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>sql</strong> (<em>string</em>) &#x2013; the sql to be executed</li>
<li><strong>bigquery_conn_id</strong> (<em>string</em>) &#x2013; reference to the BigQuery database</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.contrib.operators.bigquery_check_operator.BigQueryValueCheckOperator(sql, pass_value, tolerance=None, bigquery_conn_id=&apos;bigquery_default&apos;, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.operators.check_operator.ValueCheckOperator" title="airflow.operators.check_operator.ValueCheckOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.operators.check_operator.ValueCheckOperator</span></code></a></p>
<p>Performs a simple value check using sql code.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><strong>sql</strong> (<em>string</em>) &#x2013; the sql to be executed</td>
</tr>
</tbody>
</table>




<pre>
class airflow.contrib.operators.bigquery_check_operator.BigQueryIntervalCheckOperator(table, metrics_thresholds, date_filter_column=&apos;ds&apos;, days_back=-7, bigquery_conn_id=&apos;bigquery_default&apos;, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.operators.check_operator.IntervalCheckOperator" title="airflow.operators.check_operator.IntervalCheckOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.operators.check_operator.IntervalCheckOperator</span></code></a></p>
<p>Checks that the values of metrics given as SQL expressions are within
a certain tolerance of the ones from days_back before.</p>
<p>This method constructs a query like so</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">SELECT</span> <span class="p">{</span><span class="n">metrics_threshold_dict_key</span><span class="p">}</span> <span class="n">FROM</span> <span class="p">{</span><span class="n">table</span><span class="p">}</span>
    <span class="n">WHERE</span> <span class="p">{</span><span class="n">date_filter_column</span><span class="p">}</span><span class="o">=&lt;</span><span class="n">date</span><span class="o">&gt;</span>
</pre>
</div>
</div>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>table</strong> (<em>str</em>) &#x2013; the table name</li>
<li><strong>days_back</strong> (<em>int</em>) &#x2013; number of days between ds and the ds we want to check
against. Defaults to 7 days</li>
<li><strong>metrics_threshold</strong> (<em>dict</em>) &#x2013; a dictionary of ratios indexed by metrics, for
example &#x2018;COUNT(*)&#x2019;: 1.5 would require a 50 percent or less difference
between the current day, and the prior days_back.</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.contrib.operators.bigquery_get_data.BigQueryGetDataOperator(dataset_id, table_id, max_results=&apos;100&apos;, selected_fields=None, bigquery_conn_id=&apos;bigquery_default&apos;, delegate_to=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Fetches the data from a BigQuery table (alternatively fetch data for selected columns)
and returns data in a python list. The number of elements in the returned list will
be equal to the number of rows fetched. Each element in the list will again be a list
where element would represent the columns values for that row.</p>
<p><strong>Example Result</strong>: <code class="docutils literal notranslate"><span class="pre">[[&apos;Tony&apos;,</span> <span class="pre">&apos;10&apos;],</span> <span class="pre">[&apos;Mike&apos;,</span> <span class="pre">&apos;20&apos;],</span> <span class="pre">[&apos;Steve&apos;,</span> <span class="pre">&apos;15&apos;]]</span></code></p>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">If you pass fields to <code class="docutils literal notranslate"><span class="pre">selected_fields</span></code> which are in different order than the
order of columns already in
BQ table, the data will still be in the order of BQ table.
For example if the BQ table has 3 columns as
<code class="docutils literal notranslate"><span class="pre">[A,B,C]</span></code> and you pass &#x2018;B,A&#x2019; in the <code class="docutils literal notranslate"><span class="pre">selected_fields</span></code>
the data would still be of the form <code class="docutils literal notranslate"><span class="pre">&apos;A,B&apos;</span></code>.</p>
</div>
<p><strong>Example</strong>:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">get_data</span> <span class="o">=</span> <span class="n">BigQueryGetDataOperator</span><span class="p">(</span>
    <span class="n">task_id</span><span class="o">=</span><span class="s1">&apos;get_data_from_bq&apos;</span><span class="p">,</span>
    <span class="n">dataset_id</span><span class="o">=</span><span class="s1">&apos;test_dataset&apos;</span><span class="p">,</span>
    <span class="n">table_id</span><span class="o">=</span><span class="s1">&apos;Transaction_partitions&apos;</span><span class="p">,</span>
    <span class="n">max_results</span><span class="o">=</span><span class="s1">&apos;100&apos;</span><span class="p">,</span>
    <span class="n">selected_fields</span><span class="o">=</span><span class="s1">&apos;DATE&apos;</span><span class="p">,</span>
    <span class="n">bigquery_conn_id</span><span class="o">=</span><span class="s1">&apos;airflow-service-account&apos;</span>
<span class="p">)</span>
</pre>
</div>
</div>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>dataset_id</strong> &#x2013; The dataset ID of the requested table. (templated)</li>
<li><strong>table_id</strong> (<em>string</em>) &#x2013; The table ID of the requested table. (templated)</li>
<li><strong>max_results</strong> (<em>string</em>) &#x2013; The maximum number of records (rows) to be fetched
from the table. (templated)</li>
<li><strong>selected_fields</strong> (<em>string</em>) &#x2013; List of fields to return (comma-separated). If
unspecified, all fields are returned.</li>
<li><strong>bigquery_conn_id</strong> (<em>string</em>) &#x2013; reference to a specific BigQuery hook.</li>
<li><strong>delegate_to</strong> (<em>string</em>) &#x2013; The account to impersonate, if any.
For this to work, the service account making the request must have domain-wide
delegation enabled.</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.contrib.operators.bigquery_operator.BigQueryCreateEmptyTableOperator(dataset_id, table_id, project_id=None, schema_fields=None, gcs_schema_object=None, time_partitioning={}, bigquery_conn_id=&apos;bigquery_default&apos;, google_cloud_storage_conn_id=&apos;google_cloud_default&apos;, delegate_to=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Creates a new, empty table in the specified BigQuery dataset,
optionally with schema.</p>
<p>The schema to be used for the BigQuery table may be specified in one of
two ways. You may either directly pass the schema fields in, or you may
point the operator to a Google cloud storage object name. The object in
Google cloud storage must be a JSON file with the schema fields in it.
You can also create a table without schema.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>project_id</strong> (<em>string</em>) &#x2013; The project to create the table into. (templated)</li>
<li><strong>dataset_id</strong> (<em>string</em>) &#x2013; The dataset to create the table into. (templated)</li>
<li><strong>table_id</strong> (<em>string</em>) &#x2013; The Name of the table to be created. (templated)</li>
<li><strong>schema_fields</strong> (<em>list</em>) &#x2013; <p>If set, the schema field list as defined here:
<a class="reference external" href="https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.load.schema">https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.load.schema</a></p>
<p><strong>Example</strong>:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">schema_fields</span><span class="o">=</span><span class="p">[{</span><span class="s2">&quot;name&quot;</span><span class="p">:</span> <span class="s2">&quot;emp_name&quot;</span><span class="p">,</span> <span class="s2">&quot;type&quot;</span><span class="p">:</span> <span class="s2">&quot;STRING&quot;</span><span class="p">,</span> <span class="s2">&quot;mode&quot;</span><span class="p">:</span> <span class="s2">&quot;REQUIRED&quot;</span><span class="p">},</span>
               <span class="p">{</span><span class="s2">&quot;name&quot;</span><span class="p">:</span> <span class="s2">&quot;salary&quot;</span><span class="p">,</span> <span class="s2">&quot;type&quot;</span><span class="p">:</span> <span class="s2">&quot;INTEGER&quot;</span><span class="p">,</span> <span class="s2">&quot;mode&quot;</span><span class="p">:</span> <span class="s2">&quot;NULLABLE&quot;</span><span class="p">}]</span>
</pre>
</div>
</div>
</li>
<li><strong>gcs_schema_object</strong> (<em>string</em>) &#x2013; Full path to the JSON file containing
schema (templated). For
example: <code class="docutils literal notranslate"><span class="pre">gs://test-bucket/dir1/dir2/employee_schema.json</span></code></li>
<li><strong>time_partitioning</strong> (<em>dict</em>) &#x2013; <p>configure optional time partitioning fields i.e.
partition by field, type and  expiration as per API specifications.</p>
<div class="admonition seealso">
<p class="first admonition-title">See also</p>
<p class="last"><a class="reference external" href="https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#timePartitioning">https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#timePartitioning</a></p>
</div>
</li>
<li><strong>bigquery_conn_id</strong> (<em>string</em>) &#x2013; Reference to a specific BigQuery hook.</li>
<li><strong>google_cloud_storage_conn_id</strong> (<em>string</em>) &#x2013; Reference to a specific Google
cloud storage hook.</li>
<li><strong>delegate_to</strong> (<em>string</em>) &#x2013; The account to impersonate, if any. For this to
work, the service account making the request must have domain-wide
delegation enabled.</li>
</ul>
</td>
</tr>
</tbody>
</table>
<p><strong>Example (with schema JSON in GCS)</strong>:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">CreateTable</span> <span class="o">=</span> <span class="n">BigQueryCreateEmptyTableOperator</span><span class="p">(</span>
    <span class="n">task_id</span><span class="o">=</span><span class="s1">&apos;BigQueryCreateEmptyTableOperator_task&apos;</span><span class="p">,</span>
    <span class="n">dataset_id</span><span class="o">=</span><span class="s1">&apos;ODS&apos;</span><span class="p">,</span>
    <span class="n">table_id</span><span class="o">=</span><span class="s1">&apos;Employees&apos;</span><span class="p">,</span>
    <span class="n">project_id</span><span class="o">=</span><span class="s1">&apos;internal-gcp-project&apos;</span><span class="p">,</span>
    <span class="n">gcs_schema_object</span><span class="o">=</span><span class="s1">&apos;gs://schema-bucket/employee_schema.json&apos;</span><span class="p">,</span>
    <span class="n">bigquery_conn_id</span><span class="o">=</span><span class="s1">&apos;airflow-service-account&apos;</span><span class="p">,</span>
    <span class="n">google_cloud_storage_conn_id</span><span class="o">=</span><span class="s1">&apos;airflow-service-account&apos;</span>
<span class="p">)</span>
</pre>
</div>
</div>
<p><strong>Corresponding Schema file</strong> (<code class="docutils literal notranslate"><span class="pre">employee_schema.json</span></code>):</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="p">[</span>
  <span class="p">{</span>
    <span class="s2">&quot;mode&quot;</span><span class="p">:</span> <span class="s2">&quot;NULLABLE&quot;</span><span class="p">,</span>
    <span class="s2">&quot;name&quot;</span><span class="p">:</span> <span class="s2">&quot;emp_name&quot;</span><span class="p">,</span>
    <span class="s2">&quot;type&quot;</span><span class="p">:</span> <span class="s2">&quot;STRING&quot;</span>
  <span class="p">},</span>
  <span class="p">{</span>
    <span class="s2">&quot;mode&quot;</span><span class="p">:</span> <span class="s2">&quot;REQUIRED&quot;</span><span class="p">,</span>
    <span class="s2">&quot;name&quot;</span><span class="p">:</span> <span class="s2">&quot;salary&quot;</span><span class="p">,</span>
    <span class="s2">&quot;type&quot;</span><span class="p">:</span> <span class="s2">&quot;INTEGER&quot;</span>
  <span class="p">}</span>
<span class="p">]</span>
</pre>
</div>
</div>
<p><strong>Example (with schema in the DAG)</strong>:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">CreateTable</span> <span class="o">=</span> <span class="n">BigQueryCreateEmptyTableOperator</span><span class="p">(</span>
    <span class="n">task_id</span><span class="o">=</span><span class="s1">&apos;BigQueryCreateEmptyTableOperator_task&apos;</span><span class="p">,</span>
    <span class="n">dataset_id</span><span class="o">=</span><span class="s1">&apos;ODS&apos;</span><span class="p">,</span>
    <span class="n">table_id</span><span class="o">=</span><span class="s1">&apos;Employees&apos;</span><span class="p">,</span>
    <span class="n">project_id</span><span class="o">=</span><span class="s1">&apos;internal-gcp-project&apos;</span><span class="p">,</span>
    <span class="n">schema_fields</span><span class="o">=</span><span class="p">[{</span><span class="s2">&quot;name&quot;</span><span class="p">:</span> <span class="s2">&quot;emp_name&quot;</span><span class="p">,</span> <span class="s2">&quot;type&quot;</span><span class="p">:</span> <span class="s2">&quot;STRING&quot;</span><span class="p">,</span> <span class="s2">&quot;mode&quot;</span><span class="p">:</span> <span class="s2">&quot;REQUIRED&quot;</span><span class="p">},</span>
                   <span class="p">{</span><span class="s2">&quot;name&quot;</span><span class="p">:</span> <span class="s2">&quot;salary&quot;</span><span class="p">,</span> <span class="s2">&quot;type&quot;</span><span class="p">:</span> <span class="s2">&quot;INTEGER&quot;</span><span class="p">,</span> <span class="s2">&quot;mode&quot;</span><span class="p">:</span> <span class="s2">&quot;NULLABLE&quot;</span><span class="p">}],</span>
    <span class="n">bigquery_conn_id</span><span class="o">=</span><span class="s1">&apos;airflow-service-account&apos;</span><span class="p">,</span>
    <span class="n">google_cloud_storage_conn_id</span><span class="o">=</span><span class="s1">&apos;airflow-service-account&apos;</span>
<span class="p">)</span>
</pre>
</div>
</div>




<pre>
class airflow.contrib.operators.bigquery_operator.BigQueryCreateExternalTableOperator(bucket, source_objects, destination_project_dataset_table, schema_fields=None, schema_object=None, source_format=&apos;CSV&apos;, compression=&apos;NONE&apos;, skip_leading_rows=0, field_delimiter=&apos;, &apos;, max_bad_records=0, quote_character=None, allow_quoted_newlines=False, allow_jagged_rows=False, bigquery_conn_id=&apos;bigquery_default&apos;, google_cloud_storage_conn_id=&apos;google_cloud_default&apos;, delegate_to=None, src_fmt_configs={}, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Creates a new external table in the dataset with the data in Google Cloud
Storage.</p>
<p>The schema to be used for the BigQuery table may be specified in one of
two ways. You may either directly pass the schema fields in, or you may
point the operator to a Google cloud storage object name. The object in
Google cloud storage must be a JSON file with the schema fields in it.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>bucket</strong> (<em>string</em>) &#x2013; The bucket to point the external table to. (templated)</li>
<li><strong>source_objects</strong> &#x2013; List of Google cloud storage URIs to point
table to. (templated)
If source_format is &#x2018;DATASTORE_BACKUP&#x2019;, the list must only contain a single URI.</li>
<li><strong>destination_project_dataset_table</strong> (<em>string</em>) &#x2013; The dotted (&lt;project&gt;.)&lt;dataset&gt;.&lt;table&gt;
BigQuery table to load data into (templated). If &lt;project&gt; is not included,
project will be the project defined in the connection json.</li>
<li><strong>schema_fields</strong> (<em>list</em>) &#x2013; <p>If set, the schema field list as defined here:
<a class="reference external" href="https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.load.schema">https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.load.schema</a></p>
<p><strong>Example</strong>:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">schema_fields</span><span class="o">=</span><span class="p">[{</span><span class="s2">&quot;name&quot;</span><span class="p">:</span> <span class="s2">&quot;emp_name&quot;</span><span class="p">,</span> <span class="s2">&quot;type&quot;</span><span class="p">:</span> <span class="s2">&quot;STRING&quot;</span><span class="p">,</span> <span class="s2">&quot;mode&quot;</span><span class="p">:</span> <span class="s2">&quot;REQUIRED&quot;</span><span class="p">},</span>
               <span class="p">{</span><span class="s2">&quot;name&quot;</span><span class="p">:</span> <span class="s2">&quot;salary&quot;</span><span class="p">,</span> <span class="s2">&quot;type&quot;</span><span class="p">:</span> <span class="s2">&quot;INTEGER&quot;</span><span class="p">,</span> <span class="s2">&quot;mode&quot;</span><span class="p">:</span> <span class="s2">&quot;NULLABLE&quot;</span><span class="p">}]</span>
</pre>
</div>
</div>
<p>Should not be set when source_format is &#x2018;DATASTORE_BACKUP&#x2019;.</p>
</li>
<li><strong>schema_object</strong> &#x2013; If set, a GCS object path pointing to a .json file that
contains the schema for the table. (templated)</li>
<li><strong>schema_object</strong> &#x2013; string</li>
<li><strong>source_format</strong> (<em>string</em>) &#x2013; File format of the data.</li>
<li><strong>compression</strong> (<em>string</em>) &#x2013; [Optional] The compression type of the data source.
Possible values include GZIP and NONE.
The default value is NONE.
This setting is ignored for Google Cloud Bigtable,
Google Cloud Datastore backups and Avro formats.</li>
<li><strong>skip_leading_rows</strong> (<em>int</em>) &#x2013; Number of rows to skip when loading from a CSV.</li>
<li><strong>field_delimiter</strong> (<em>string</em>) &#x2013; The delimiter to use for the CSV.</li>
<li><strong>max_bad_records</strong> (<em>int</em>) &#x2013; The maximum number of bad records that BigQuery can
ignore when running the job.</li>
<li><strong>quote_character</strong> (<em>string</em>) &#x2013; The value that is used to quote data sections in a CSV file.</li>
<li><strong>allow_quoted_newlines</strong> (<em>boolean</em>) &#x2013; Whether to allow quoted newlines (true) or not (false).</li>
<li><strong>allow_jagged_rows</strong> (<em>bool</em>) &#x2013; Accept rows that are missing trailing optional columns.
The missing values are treated as nulls. If false, records with missing trailing
columns are treated as bad records, and if there are too many bad records, an
invalid error is returned in the job result. Only applicable to CSV, ignored
for other formats.</li>
<li><strong>bigquery_conn_id</strong> (<em>string</em>) &#x2013; Reference to a specific BigQuery hook.</li>
<li><strong>google_cloud_storage_conn_id</strong> (<em>string</em>) &#x2013; Reference to a specific Google
cloud storage hook.</li>
<li><strong>delegate_to</strong> (<em>string</em>) &#x2013; The account to impersonate, if any. For this to
work, the service account making the request must have domain-wide
delegation enabled.</li>
<li><strong>src_fmt_configs</strong> (<em>dict</em>) &#x2013; configure optional fields specific to the source format</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.contrib.operators.bigquery_operator.BigQueryOperator(bql=None, sql=None, destination_dataset_table=False, write_disposition=&apos;WRITE_EMPTY&apos;, allow_large_results=False, flatten_results=False, bigquery_conn_id=&apos;bigquery_default&apos;, delegate_to=None, udf_config=False, use_legacy_sql=True, maximum_billing_tier=None, maximum_bytes_billed=None, create_disposition=&apos;CREATE_IF_NEEDED&apos;, schema_update_options=(), query_params=None, priority=&apos;INTERACTIVE&apos;, time_partitioning={}, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Executes BigQuery SQL queries in a specific BigQuery database</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>bql</strong> (<em>Can receive a str representing a sql statement</em><em>,
</em><em>a list of str</em><em> (</em><em>sql statements</em><em>)</em><em>, or </em><em>reference to a template file.
Template reference are recognized by str ending in &apos;.sql&apos;.</em>) &#x2013; (Deprecated. Use <cite>sql</cite> parameter instead) the sql code to be
executed (templated)</li>
<li><strong>sql</strong> (<em>Can receive a str representing a sql statement</em><em>,
</em><em>a list of str</em><em> (</em><em>sql statements</em><em>)</em><em>, or </em><em>reference to a template file.
Template reference are recognized by str ending in &apos;.sql&apos;.</em>) &#x2013; the sql code to be executed (templated)</li>
<li><strong>destination_dataset_table</strong> (<em>string</em>) &#x2013; A dotted
(&lt;project&gt;.|&lt;project&gt;:)&lt;dataset&gt;.&lt;table&gt; that, if set, will store the results
of the query. (templated)</li>
<li><strong>write_disposition</strong> (<em>string</em>) &#x2013; Specifies the action that occurs if the destination table
already exists. (default: &#x2018;WRITE_EMPTY&#x2019;)</li>
<li><strong>create_disposition</strong> (<em>string</em>) &#x2013; Specifies whether the job is allowed to create new tables.
(default: &#x2018;CREATE_IF_NEEDED&#x2019;)</li>
<li><strong>allow_large_results</strong> (<em>boolean</em>) &#x2013; Whether to allow large results.</li>
<li><strong>flatten_results</strong> (<em>boolean</em>) &#x2013; If true and query uses legacy SQL dialect, flattens
all nested and repeated fields in the query results. <code class="docutils literal notranslate"><span class="pre">allow_large_results</span></code>
must be <code class="docutils literal notranslate"><span class="pre">true</span></code> if this is set to <code class="docutils literal notranslate"><span class="pre">false</span></code>. For standard SQL queries, this
flag is ignored and results are never flattened.</li>
<li><strong>bigquery_conn_id</strong> (<em>string</em>) &#x2013; reference to a specific BigQuery hook.</li>
<li><strong>delegate_to</strong> (<em>string</em>) &#x2013; The account to impersonate, if any.
For this to work, the service account making the request must have domain-wide
delegation enabled.</li>
<li><strong>udf_config</strong> (<em>list</em>) &#x2013; The User Defined Function configuration for the query.
See <a class="reference external" href="https://cloud.google.com/bigquery/user-defined-functions">https://cloud.google.com/bigquery/user-defined-functions</a> for details.</li>
<li><strong>use_legacy_sql</strong> (<em>boolean</em>) &#x2013; Whether to use legacy SQL (true) or standard SQL (false).</li>
<li><strong>maximum_billing_tier</strong> (<em>integer</em>) &#x2013; Positive integer that serves as a multiplier
of the basic price.
Defaults to None, in which case it uses the value set in the project.</li>
<li><strong>maximum_bytes_billed</strong> (<em>float</em>) &#x2013; Limits the bytes billed for this job.
Queries that will have bytes billed beyond this limit will fail
(without incurring a charge). If unspecified, this will be
set to your project default.</li>
<li><strong>schema_update_options</strong> (<em>tuple</em>) &#x2013; Allows the schema of the destination
table to be updated as a side effect of the load job.</li>
<li><strong>query_params</strong> (<em>dict</em>) &#x2013; a dictionary containing query parameter types and
values, passed to BigQuery.</li>
<li><strong>priority</strong> (<em>string</em>) &#x2013; Specifies a priority for the query.
Possible values include INTERACTIVE and BATCH.
The default value is INTERACTIVE.</li>
<li><strong>time_partitioning</strong> (<em>dict</em>) &#x2013; configure optional time partitioning fields i.e.
partition by field, type and
expiration as per API specifications. Note that &#x2018;field&#x2019; is not available in
conjunction with dataset.table$partition.</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.contrib.operators.bigquery_table_delete_operator.BigQueryTableDeleteOperator(deletion_dataset_table, bigquery_conn_id=&apos;bigquery_default&apos;, delegate_to=None, ignore_if_missing=False, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Deletes BigQuery tables</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>deletion_dataset_table</strong> (<em>string</em>) &#x2013; A dotted
(&lt;project&gt;.|&lt;project&gt;:)&lt;dataset&gt;.&lt;table&gt; that indicates which table
will be deleted. (templated)</li>
<li><strong>bigquery_conn_id</strong> (<em>string</em>) &#x2013; reference to a specific BigQuery hook.</li>
<li><strong>delegate_to</strong> (<em>string</em>) &#x2013; The account to impersonate, if any.
For this to work, the service account making the request must have domain-wide
delegation enabled.</li>
<li><strong>ignore_if_missing</strong> (<em>boolean</em>) &#x2013; if True, then return success even if the
requested table does not exist.</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.contrib.operators.bigquery_to_bigquery.BigQueryToBigQueryOperator(source_project_dataset_tables, destination_project_dataset_table, write_disposition=&apos;WRITE_EMPTY&apos;, create_disposition=&apos;CREATE_IF_NEEDED&apos;, bigquery_conn_id=&apos;bigquery_default&apos;, delegate_to=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Copies data from one BigQuery table to another.</p>
<div class="admonition seealso">
<p class="first admonition-title">See also</p>
<p class="last">For more details about these parameters:
<a class="reference external" href="https://cloud.google.com/bigquery/docs/reference/v2/jobs#configuration.copy">https://cloud.google.com/bigquery/docs/reference/v2/jobs#configuration.copy</a></p>
</div>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>source_project_dataset_tables</strong> (<em>list|string</em>) &#x2013; One or more
dotted (project:<a href="#id5"><span class="problematic" id="id6">|</span></a>project.)&lt;dataset&gt;.&lt;table&gt; BigQuery tables to use as the
source data. If &lt;project&gt; is not included, project will be the
project defined in the connection json. Use a list if there are multiple
source tables. (templated)</li>
<li><strong>destination_project_dataset_table</strong> (<em>string</em>) &#x2013; The destination BigQuery
table. Format is: (project:<a href="#id7"><span class="problematic" id="id8">|</span></a>project.)&lt;dataset&gt;.&lt;table&gt; (templated)</li>
<li><strong>write_disposition</strong> (<em>string</em>) &#x2013; The write disposition if the table already exists.</li>
<li><strong>create_disposition</strong> (<em>string</em>) &#x2013; The create disposition if the table doesn&#x2019;t exist.</li>
<li><strong>bigquery_conn_id</strong> (<em>string</em>) &#x2013; reference to a specific BigQuery hook.</li>
<li><strong>delegate_to</strong> (<em>string</em>) &#x2013; The account to impersonate, if any.
For this to work, the service account making the request must have domain-wide
delegation enabled.</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.contrib.operators.bigquery_to_gcs.BigQueryToCloudStorageOperator(source_project_dataset_table, destination_cloud_storage_uris, compression=&apos;NONE&apos;, export_format=&apos;CSV&apos;, field_delimiter=&apos;, &apos;, print_header=True, bigquery_conn_id=&apos;bigquery_default&apos;, delegate_to=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Transfers a BigQuery table to a Google Cloud Storage bucket.</p>
<div class="admonition seealso">
<p class="first admonition-title">See also</p>
<p class="last">For more details about these parameters:
<a class="reference external" href="https://cloud.google.com/bigquery/docs/reference/v2/jobs">https://cloud.google.com/bigquery/docs/reference/v2/jobs</a></p>
</div>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>source_project_dataset_table</strong> (<em>string</em>) &#x2013; The dotted
(&lt;project&gt;.|&lt;project&gt;:)&lt;dataset&gt;.&lt;table&gt; BigQuery table to use as the source
data. If &lt;project&gt; is not included, project will be the project
defined in the connection json. (templated)</li>
<li><strong>destination_cloud_storage_uris</strong> (<em>list</em>) &#x2013; The destination Google Cloud
Storage URI (e.g. gs://some-bucket/some-file.txt). (templated) Follows
convention defined here:
https://cloud.google.com/bigquery/exporting-data-from-bigquery#exportingmultiple</li>
<li><strong>compression</strong> (<em>string</em>) &#x2013; Type of compression to use.</li>
<li><strong>export_format</strong> &#x2013; File format to export.</li>
<li><strong>field_delimiter</strong> (<em>string</em>) &#x2013; The delimiter to use when extracting to a CSV.</li>
<li><strong>print_header</strong> (<em>boolean</em>) &#x2013; Whether to print a header for a CSV file extract.</li>
<li><strong>bigquery_conn_id</strong> (<em>string</em>) &#x2013; reference to a specific BigQuery hook.</li>
<li><strong>delegate_to</strong> (<em>string</em>) &#x2013; The account to impersonate, if any.
For this to work, the service account making the request must have domain-wide
delegation enabled.</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.contrib.operators.cassandra_to_gcs.CassandraToGoogleCloudStorageOperator(cql, bucket, filename, schema_filename=None, approx_max_file_size_bytes=1900000000, cassandra_conn_id=u&apos;cassandra_default&apos;, google_cloud_storage_conn_id=u&apos;google_cloud_default&apos;, delegate_to=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Copy data from Cassandra to Google cloud storage in JSON format</p>
<p>Note: Arrays of arrays are not supported.</p>

<pre>
classmethod convert_map_type(name, value)</pre>
<p>Converts a map to a repeated RECORD that contains two fields: &#x2018;key&#x2019; and &#x2018;value&#x2019;,
each will be converted to its corresopnding data type in BQ.</p>




<pre>
classmethod convert_tuple_type(name, value)</pre>
<p>Converts a tuple to RECORD that contains n fields, each will be converted
to its corresponding data type in bq and will be named &#x2018;field_&lt;index&gt;&#x2019;, where
index is determined by the order of the tuple elments defined in cassandra.</p>




<pre>
classmethod convert_user_type(name, value)</pre>
<p>Converts a user type to RECORD that contains n fields, where n is the
number of attributes. Each element in the user type class will be converted to its
corresponding data type in BQ.</p>







<pre>
class airflow.contrib.operators.databricks_operator.DatabricksSubmitRunOperator(json=None, spark_jar_task=None, notebook_task=None, new_cluster=None, existing_cluster_id=None, libraries=None, run_name=None, timeout_seconds=None, databricks_conn_id=&apos;databricks_default&apos;, polling_period_seconds=30, databricks_retry_limit=3, do_xcom_push=False, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Submits an Spark job run to Databricks using the
<a class="reference external" href="https://docs.databricks.com/api/latest/jobs.html#runs-submit">api/2.0/jobs/runs/submit</a>
API endpoint.</p>
<p>There are two ways to instantiate this operator.</p>
<p>In the first way, you can take the JSON payload that you typically use
to call the <code class="docutils literal notranslate"><span class="pre">api/2.0/jobs/runs/submit</span></code> endpoint and pass it directly
to our <code class="docutils literal notranslate"><span class="pre">DatabricksSubmitRunOperator</span></code> through the <code class="docutils literal notranslate"><span class="pre">json</span></code> parameter.
For example</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">json</span> <span class="o">=</span> <span class="p">{</span>
  <span class="s1">&apos;new_cluster&apos;</span><span class="p">:</span> <span class="p">{</span>
    <span class="s1">&apos;spark_version&apos;</span><span class="p">:</span> <span class="s1">&apos;2.1.0-db3-scala2.11&apos;</span><span class="p">,</span>
    <span class="s1">&apos;num_workers&apos;</span><span class="p">:</span> <span class="mi">2</span>
  <span class="p">},</span>
  <span class="s1">&apos;notebook_task&apos;</span><span class="p">:</span> <span class="p">{</span>
    <span class="s1">&apos;notebook_path&apos;</span><span class="p">:</span> <span class="s1">&apos;/Users/airflow@example.com/PrepareData&apos;</span><span class="p">,</span>
  <span class="p">},</span>
<span class="p">}</span>
<span class="n">notebook_run</span> <span class="o">=</span> <span class="n">DatabricksSubmitRunOperator</span><span class="p">(</span><span class="n">task_id</span><span class="o">=</span><span class="s1">&apos;notebook_run&apos;</span><span class="p">,</span> <span class="n">json</span><span class="o">=</span><span class="n">json</span><span class="p">)</span>
</pre>
</div>
</div>
<p>Another way to accomplish the same thing is to use the named parameters
of the <code class="docutils literal notranslate"><span class="pre">DatabricksSubmitRunOperator</span></code> directly. Note that there is exactly
one named parameter for each top level parameter in the <code class="docutils literal notranslate"><span class="pre">runs/submit</span></code>
endpoint. In this method, your code would look like this:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">new_cluster</span> <span class="o">=</span> <span class="p">{</span>
  <span class="s1">&apos;spark_version&apos;</span><span class="p">:</span> <span class="s1">&apos;2.1.0-db3-scala2.11&apos;</span><span class="p">,</span>
  <span class="s1">&apos;num_workers&apos;</span><span class="p">:</span> <span class="mi">2</span>
<span class="p">}</span>
<span class="n">notebook_task</span> <span class="o">=</span> <span class="p">{</span>
  <span class="s1">&apos;notebook_path&apos;</span><span class="p">:</span> <span class="s1">&apos;/Users/airflow@example.com/PrepareData&apos;</span><span class="p">,</span>
<span class="p">}</span>
<span class="n">notebook_run</span> <span class="o">=</span> <span class="n">DatabricksSubmitRunOperator</span><span class="p">(</span>
    <span class="n">task_id</span><span class="o">=</span><span class="s1">&apos;notebook_run&apos;</span><span class="p">,</span>
    <span class="n">new_cluster</span><span class="o">=</span><span class="n">new_cluster</span><span class="p">,</span>
    <span class="n">notebook_task</span><span class="o">=</span><span class="n">notebook_task</span><span class="p">)</span>
</pre>
</div>
</div>
<p>In the case where both the json parameter <strong>AND</strong> the named parameters
are provided, they will be merged together. If there are conflicts during the merge,
the named parameters will take precedence and override the top level <code class="docutils literal notranslate"><span class="pre">json</span></code> keys.</p>

<pre>Currently the named parameters that DatabricksSubmitRunOperator supports are</pre>
<ul class="first last simple">
<li><code class="docutils literal notranslate"><span class="pre">spark_jar_task</span></code></li>
<li><code class="docutils literal notranslate"><span class="pre">notebook_task</span></code></li>
<li><code class="docutils literal notranslate"><span class="pre">new_cluster</span></code></li>
<li><code class="docutils literal notranslate"><span class="pre">existing_cluster_id</span></code></li>
<li><code class="docutils literal notranslate"><span class="pre">libraries</span></code></li>
<li><code class="docutils literal notranslate"><span class="pre">run_name</span></code></li>
<li><code class="docutils literal notranslate"><span class="pre">timeout_seconds</span></code></li>
</ul>


<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>json</strong> (<em>dict</em>) &#x2013; <p>A JSON object containing API parameters which will be passed
directly to the <code class="docutils literal notranslate"><span class="pre">api/2.0/jobs/runs/submit</span></code> endpoint. The other named parameters
(i.e. <code class="docutils literal notranslate"><span class="pre">spark_jar_task</span></code>, <code class="docutils literal notranslate"><span class="pre">notebook_task</span></code>..) to this operator will
be merged with this json dictionary if they are provided.
If there are conflicts during the merge, the named parameters will
take precedence and override the top level json keys. (templated)</p>
<div class="admonition seealso">
<p class="first admonition-title">See also</p>
<p class="last">For more information about templating see <a class="reference internal" href="concepts.html#jinja-templating"><span class="std std-ref">Jinja Templating</span></a>.
<a class="reference external" href="https://docs.databricks.com/api/latest/jobs.html#runs-submit">https://docs.databricks.com/api/latest/jobs.html#runs-submit</a></p>
</div>
</li>
<li><strong>spark_jar_task</strong> (<em>dict</em>) &#x2013; <p>The main class and parameters for the JAR task. Note that
the actual JAR is specified in the <code class="docutils literal notranslate"><span class="pre">libraries</span></code>.
<em>EITHER</em> <code class="docutils literal notranslate"><span class="pre">spark_jar_task</span></code> <em>OR</em> <code class="docutils literal notranslate"><span class="pre">notebook_task</span></code> should be specified.
This field will be templated.</p>
<div class="admonition seealso">
<p class="first admonition-title">See also</p>
<p class="last"><a class="reference external" href="https://docs.databricks.com/api/latest/jobs.html#jobssparkjartask">https://docs.databricks.com/api/latest/jobs.html#jobssparkjartask</a></p>
</div>
</li>
<li><strong>notebook_task</strong> (<em>dict</em>) &#x2013; <p>The notebook path and parameters for the notebook task.
<em>EITHER</em> <code class="docutils literal notranslate"><span class="pre">spark_jar_task</span></code> <em>OR</em> <code class="docutils literal notranslate"><span class="pre">notebook_task</span></code> should be specified.
This field will be templated.</p>
<div class="admonition seealso">
<p class="first admonition-title">See also</p>
<p class="last"><a class="reference external" href="https://docs.databricks.com/api/latest/jobs.html#jobsnotebooktask">https://docs.databricks.com/api/latest/jobs.html#jobsnotebooktask</a></p>
</div>
</li>
<li><strong>new_cluster</strong> (<em>dict</em>) &#x2013; <p>Specs for a new cluster on which this task will be run.
<em>EITHER</em> <code class="docutils literal notranslate"><span class="pre">new_cluster</span></code> <em>OR</em> <code class="docutils literal notranslate"><span class="pre">existing_cluster_id</span></code> should be specified.
This field will be templated.</p>
<div class="admonition seealso">
<p class="first admonition-title">See also</p>
<p class="last"><a class="reference external" href="https://docs.databricks.com/api/latest/jobs.html#jobsclusterspecnewcluster">https://docs.databricks.com/api/latest/jobs.html#jobsclusterspecnewcluster</a></p>
</div>
</li>
<li><strong>existing_cluster_id</strong> (<em>string</em>) &#x2013; ID for existing cluster on which to run this task.
<em>EITHER</em> <code class="docutils literal notranslate"><span class="pre">new_cluster</span></code> <em>OR</em> <code class="docutils literal notranslate"><span class="pre">existing_cluster_id</span></code> should be specified.
This field will be templated.</li>
<li><strong>libraries</strong> (<em>list of dicts</em>) &#x2013; <p>Libraries which this run will use.
This field will be templated.</p>
<div class="admonition seealso">
<p class="first admonition-title">See also</p>
<p class="last"><a class="reference external" href="https://docs.databricks.com/api/latest/libraries.html#managedlibrarieslibrary">https://docs.databricks.com/api/latest/libraries.html#managedlibrarieslibrary</a></p>
</div>
</li>
<li><strong>run_name</strong> (<em>string</em>) &#x2013; The run name used for this task.
By default this will be set to the Airflow <code class="docutils literal notranslate"><span class="pre">task_id</span></code>. This <code class="docutils literal notranslate"><span class="pre">task_id</span></code> is a
required parameter of the superclass <code class="docutils literal notranslate"><span class="pre">BaseOperator</span></code>.
This field will be templated.</li>
<li><strong>timeout_seconds</strong> (<em>int32</em>) &#x2013; The timeout for this run. By default a value of 0 is used
which means to have no timeout.
This field will be templated.</li>
<li><strong>databricks_conn_id</strong> (<em>string</em>) &#x2013; The name of the Airflow connection to use.
By default and in the common case this will be <code class="docutils literal notranslate"><span class="pre">databricks_default</span></code>. To use
token based authentication, provide the key <code class="docutils literal notranslate"><span class="pre">token</span></code> in the extra field for the
connection.</li>
<li><strong>polling_period_seconds</strong> (<em>int</em>) &#x2013; Controls the rate which we poll for the result of
this run. By default the operator will poll every 30 seconds.</li>
<li><strong>databricks_retry_limit</strong> (<em>int</em>) &#x2013; Amount of times retry if the Databricks backend is
unreachable. Its value must be greater than or equal to 1.</li>
<li><strong>do_xcom_push</strong> (<em>boolean</em>) &#x2013; Whether we should push run_id and run_page_url to xcom.</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.contrib.operators.dataflow_operator.DataFlowJavaOperator(jar, dataflow_default_options=None, options=None, gcp_conn_id=&apos;google_cloud_default&apos;, delegate_to=None, poll_sleep=10, job_class=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Start a Java Cloud DataFlow batch job. The parameters of the operation
will be passed to the job.</p>
<p>It&#x2019;s a good practice to define dataflow_* parameters in the default_args of the dag
like the project, zone and staging location.</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">default_args</span> <span class="o">=</span> <span class="p">{</span>
    <span class="s1">&apos;dataflow_default_options&apos;</span><span class="p">:</span> <span class="p">{</span>
        <span class="s1">&apos;project&apos;</span><span class="p">:</span> <span class="s1">&apos;my-gcp-project&apos;</span><span class="p">,</span>
        <span class="s1">&apos;zone&apos;</span><span class="p">:</span> <span class="s1">&apos;europe-west1-d&apos;</span><span class="p">,</span>
        <span class="s1">&apos;stagingLocation&apos;</span><span class="p">:</span> <span class="s1">&apos;gs://my-staging-bucket/staging/&apos;</span>
    <span class="p">}</span>
<span class="p">}</span>
</pre>
</div>
</div>
<p>You need to pass the path to your dataflow as a file reference with the <code class="docutils literal notranslate"><span class="pre">jar</span></code>
parameter, the jar needs to be a self executing jar (see documentation here:
<a class="reference external" href="https://beam.apache.org/documentation/runners/dataflow/#self-executing-jar">https://beam.apache.org/documentation/runners/dataflow/#self-executing-jar</a>).
Use <code class="docutils literal notranslate"><span class="pre">options</span></code> to pass on options to your job.</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">t1</span> <span class="o">=</span> <span class="n">DataFlowOperation</span><span class="p">(</span>
    <span class="n">task_id</span><span class="o">=</span><span class="s1">&apos;datapflow_example&apos;</span><span class="p">,</span>
    <span class="n">jar</span><span class="o">=</span><span class="s1">&apos;{{var.value.gcp_dataflow_base}}pipeline/build/libs/pipeline-example-1.0.jar&apos;</span><span class="p">,</span>
    <span class="n">options</span><span class="o">=</span><span class="p">{</span>
        <span class="s1">&apos;autoscalingAlgorithm&apos;</span><span class="p">:</span> <span class="s1">&apos;BASIC&apos;</span><span class="p">,</span>
        <span class="s1">&apos;maxNumWorkers&apos;</span><span class="p">:</span> <span class="s1">&apos;50&apos;</span><span class="p">,</span>
        <span class="s1">&apos;start&apos;</span><span class="p">:</span> <span class="s1">&apos;{{ds}}&apos;</span><span class="p">,</span>
        <span class="s1">&apos;partitionType&apos;</span><span class="p">:</span> <span class="s1">&apos;DAY&apos;</span><span class="p">,</span>
        <span class="s1">&apos;labels&apos;</span><span class="p">:</span> <span class="p">{</span><span class="s1">&apos;foo&apos;</span> <span class="p">:</span> <span class="s1">&apos;bar&apos;</span><span class="p">}</span>
    <span class="p">},</span>
    <span class="n">gcp_conn_id</span><span class="o">=</span><span class="s1">&apos;gcp-airflow-service-account&apos;</span><span class="p">,</span>
    <span class="n">dag</span><span class="o">=</span><span class="n">my</span><span class="o">-</span><span class="n">dag</span><span class="p">)</span>
</pre>
</div>
</div>
<p>Both <code class="docutils literal notranslate"><span class="pre">jar</span></code> and <code class="docutils literal notranslate"><span class="pre">options</span></code> are templated so you can use variables in them.</p>




<pre>
class airflow.contrib.operators.dataflow_operator.DataflowTemplateOperator(template, dataflow_default_options=None, parameters=None, gcp_conn_id=&apos;google_cloud_default&apos;, delegate_to=None, poll_sleep=10, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Start a Templated Cloud DataFlow batch job. The parameters of the operation
will be passed to the job.
It&#x2019;s a good practice to define dataflow_* parameters in the default_args of the dag
like the project, zone and staging location.</p>
<div class="admonition seealso">
<p class="first admonition-title">See also</p>
<p class="last"><a class="reference external" href="https://cloud.google.com/dataflow/docs/reference/rest/v1b3/LaunchTemplateParameters">https://cloud.google.com/dataflow/docs/reference/rest/v1b3/LaunchTemplateParameters</a>
<a class="reference external" href="https://cloud.google.com/dataflow/docs/reference/rest/v1b3/RuntimeEnvironment">https://cloud.google.com/dataflow/docs/reference/rest/v1b3/RuntimeEnvironment</a></p>
</div>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">default_args</span> <span class="o">=</span> <span class="p">{</span>
    <span class="s1">&apos;dataflow_default_options&apos;</span><span class="p">:</span> <span class="p">{</span>
        <span class="s1">&apos;project&apos;</span><span class="p">:</span> <span class="s1">&apos;my-gcp-project&apos;</span>
        <span class="s1">&apos;zone&apos;</span><span class="p">:</span> <span class="s1">&apos;europe-west1-d&apos;</span><span class="p">,</span>
        <span class="s1">&apos;tempLocation&apos;</span><span class="p">:</span> <span class="s1">&apos;gs://my-staging-bucket/staging/&apos;</span>
        <span class="p">}</span>
    <span class="p">}</span>
<span class="p">}</span>
</pre>
</div>
</div>
<p>You need to pass the path to your dataflow template as a file reference with the
<code class="docutils literal notranslate"><span class="pre">template</span></code> parameter. Use <code class="docutils literal notranslate"><span class="pre">parameters</span></code> to pass on parameters to your job.
Use <code class="docutils literal notranslate"><span class="pre">environment</span></code> to pass on runtime environment variables to your job.</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">t1</span> <span class="o">=</span> <span class="n">DataflowTemplateOperator</span><span class="p">(</span>
    <span class="n">task_id</span><span class="o">=</span><span class="s1">&apos;datapflow_example&apos;</span><span class="p">,</span>
    <span class="n">template</span><span class="o">=</span><span class="s1">&apos;{{var.value.gcp_dataflow_base}}&apos;</span><span class="p">,</span>
    <span class="n">parameters</span><span class="o">=</span><span class="p">{</span>
        <span class="s1">&apos;inputFile&apos;</span><span class="p">:</span> <span class="s2">&quot;gs://bucket/input/my_input.txt&quot;</span><span class="p">,</span>
        <span class="s1">&apos;outputFile&apos;</span><span class="p">:</span> <span class="s2">&quot;gs://bucket/output/my_output.txt&quot;</span>
    <span class="p">},</span>
    <span class="n">gcp_conn_id</span><span class="o">=</span><span class="s1">&apos;gcp-airflow-service-account&apos;</span><span class="p">,</span>
    <span class="n">dag</span><span class="o">=</span><span class="n">my</span><span class="o">-</span><span class="n">dag</span><span class="p">)</span>
</pre>
</div>
</div>
<p><code class="docutils literal notranslate"><span class="pre">template</span></code>, <code class="docutils literal notranslate"><span class="pre">dataflow_default_options</span></code> and <code class="docutils literal notranslate"><span class="pre">parameters</span></code> are templated so you can
use variables in them.</p>




<pre>
class airflow.contrib.operators.dataflow_operator.DataFlowPythonOperator(py_file, py_options=None, dataflow_default_options=None, options=None, gcp_conn_id=&apos;google_cloud_default&apos;, delegate_to=None, poll_sleep=10, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>

<pre>
execute(context)</pre>
<p>Execute the python dataflow job.</p>







<pre>
class airflow.contrib.operators.dataproc_operator.DataprocClusterCreateOperator(cluster_name, project_id, num_workers, zone, network_uri=None, subnetwork_uri=None, internal_ip_only=None, tags=None, storage_bucket=None, init_actions_uris=None, init_action_timeout=&apos;10m&apos;, metadata=None, image_version=None, properties=None, master_machine_type=&apos;n1-standard-4&apos;, master_disk_size=500, worker_machine_type=&apos;n1-standard-4&apos;, worker_disk_size=500, num_preemptible_workers=0, labels=None, region=&apos;global&apos;, gcp_conn_id=&apos;google_cloud_default&apos;, delegate_to=None, service_account=None, service_account_scopes=None, idle_delete_ttl=None, auto_delete_time=None, auto_delete_ttl=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Create a new cluster on Google Cloud Dataproc. The operator will wait until the
creation is successful or an error occurs in the creation process.</p>
<p>The parameters allow to configure the cluster. Please refer to</p>
<p><a class="reference external" href="https://cloud.google.com/dataproc/docs/reference/rest/v1/projects.regions.clusters">https://cloud.google.com/dataproc/docs/reference/rest/v1/projects.regions.clusters</a></p>
<p>for a detailed explanation on the different parameters. Most of the configuration
parameters detailed in the link are available as a parameter to this operator.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>cluster_name</strong> (<em>string</em>) &#x2013; The name of the DataProc cluster to create. (templated)</li>
<li><strong>project_id</strong> (<em>string</em>) &#x2013; The ID of the google cloud project in which
to create the cluster. (templated)</li>
<li><strong>num_workers</strong> (<em>int</em>) &#x2013; The # of workers to spin up</li>
<li><strong>storage_bucket</strong> (<em>string</em>) &#x2013; The storage bucket to use, setting to None lets dataproc
generate a custom one for you</li>
<li><strong>init_actions_uris</strong> (<em>list</em><em>[</em><em>string</em><em>]</em>) &#x2013; List of GCS uri&#x2019;s containing
dataproc initialization scripts</li>
<li><strong>init_action_timeout</strong> (<em>string</em>) &#x2013; Amount of time executable scripts in
init_actions_uris has to complete</li>
<li><strong>metadata</strong> (<em>dict</em>) &#x2013; dict of key-value google compute engine metadata entries
to add to all instances</li>
<li><strong>image_version</strong> (<em>string</em>) &#x2013; the version of software inside the Dataproc cluster</li>
<li><strong>properties</strong> (<em>dict</em>) &#x2013; dict of properties to set on
config files (e.g. spark-defaults.conf), see
<a class="reference external" href="https://cloud.google.com/dataproc/docs/reference/rest/v1/">https://cloud.google.com/dataproc/docs/reference/rest/v1/</a>         projects.regions.clusters#SoftwareConfig</li>
<li><strong>master_machine_type</strong> (<em>string</em>) &#x2013; Compute engine machine type to use for the master node</li>
<li><strong>master_disk_size</strong> (<em>int</em>) &#x2013; Disk size for the master node</li>
<li><strong>worker_machine_type</strong> (<em>string</em>) &#x2013; Compute engine machine type to use for the worker nodes</li>
<li><strong>worker_disk_size</strong> (<em>int</em>) &#x2013; Disk size for the worker nodes</li>
<li><strong>num_preemptible_workers</strong> (<em>int</em>) &#x2013; The # of preemptible worker nodes to spin up</li>
<li><strong>labels</strong> (<em>dict</em>) &#x2013; dict of labels to add to the cluster</li>
<li><strong>zone</strong> (<em>string</em>) &#x2013; The zone where the cluster will be located. (templated)</li>
<li><strong>network_uri</strong> (<em>string</em>) &#x2013; The network uri to be used for machine communication, cannot be
specified with subnetwork_uri</li>
<li><strong>subnetwork_uri</strong> (<em>string</em>) &#x2013; The subnetwork uri to be used for machine communication,
cannot be specified with network_uri</li>
<li><strong>internal_ip_only</strong> (<em>bool</em>) &#x2013; If true, all instances in the cluster will only
have internal IP addresses. This can only be enabled for subnetwork
enabled networks</li>
<li><strong>tags</strong> (<em>list</em><em>[</em><em>string</em><em>]</em>) &#x2013; The GCE tags to add to all instances</li>
<li><strong>region</strong> &#x2013; leave as &#x2018;global&#x2019;, might become relevant in the future. (templated)</li>
<li><strong>gcp_conn_id</strong> (<em>string</em>) &#x2013; The connection ID to use connecting to Google Cloud Platform.</li>
<li><strong>delegate_to</strong> (<em>string</em>) &#x2013; The account to impersonate, if any.
For this to work, the service account making the request must have domain-wide
delegation enabled.</li>
<li><strong>service_account</strong> (<em>string</em>) &#x2013; The service account of the dataproc instances.</li>
<li><strong>service_account_scopes</strong> (<em>list</em><em>[</em><em>string</em><em>]</em>) &#x2013; The URIs of service account scopes to be included.</li>
<li><strong>idle_delete_ttl</strong> (<em>int</em>) &#x2013; The longest duration that cluster would keep alive while
staying idle. Passing this threshold will cause cluster to be auto-deleted.
A duration in seconds.</li>
<li><strong>auto_delete_time</strong> (<em>datetime.datetime</em>) &#x2013; The time when cluster will be auto-deleted.</li>
<li><strong>auto_delete_ttl</strong> (<em>int</em>) &#x2013; The life duration of cluster, the cluster will be
auto-deleted at the end of this duration.
A duration in seconds. (If auto_delete_time is set this parameter will be ignored)</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.contrib.operators.dataproc_operator.DataprocClusterScaleOperator(cluster_name, project_id, region=&apos;global&apos;, gcp_conn_id=&apos;google_cloud_default&apos;, delegate_to=None, num_workers=2, num_preemptible_workers=0, graceful_decommission_timeout=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Scale, up or down, a cluster on Google Cloud Dataproc.
The operator will wait until the cluster is re-scaled.</p>
<p><strong>Example</strong>:</p>

<pre>t1 = DataprocClusterScaleOperator(</pre>
task_id=&#x2019;dataproc_scale&#x2019;,
project_id=&#x2019;my-project&#x2019;,
cluster_name=&#x2019;cluster-1&#x2019;,
num_workers=10,
num_preemptible_workers=10,
graceful_decommission_timeout=&#x2018;1h&#x2019;
dag=dag)

<div class="admonition seealso">
<p class="first admonition-title">See also</p>
<p class="last">For more detail on about scaling clusters have a look at the reference:
<a class="reference external" href="https://cloud.google.com/dataproc/docs/concepts/configuring-clusters/scaling-clusters">https://cloud.google.com/dataproc/docs/concepts/configuring-clusters/scaling-clusters</a></p>
</div>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>cluster_name</strong> (<em>string</em>) &#x2013; The name of the cluster to scale. (templated)</li>
<li><strong>project_id</strong> (<em>string</em>) &#x2013; The ID of the google cloud project in which
the cluster runs. (templated)</li>
<li><strong>region</strong> (<em>string</em>) &#x2013; The region for the dataproc cluster. (templated)</li>
<li><strong>gcp_conn_id</strong> (<em>string</em>) &#x2013; The connection ID to use connecting to Google Cloud Platform.</li>
<li><strong>num_workers</strong> (<em>int</em>) &#x2013; The new number of workers</li>
<li><strong>num_preemptible_workers</strong> (<em>int</em>) &#x2013; The new number of preemptible workers</li>
<li><strong>graceful_decommission_timeout</strong> (<em>string</em>) &#x2013; Timeout for graceful YARN decomissioning.
Maximum value is 1d</li>
<li><strong>delegate_to</strong> (<em>string</em>) &#x2013; The account to impersonate, if any.
For this to work, the service account making the request must have domain-wide
delegation enabled.</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.contrib.operators.dataproc_operator.DataprocClusterDeleteOperator(cluster_name, project_id, region=&apos;global&apos;, gcp_conn_id=&apos;google_cloud_default&apos;, delegate_to=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Delete a cluster on Google Cloud Dataproc. The operator will wait until the
cluster is destroyed.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>cluster_name</strong> (<em>string</em>) &#x2013; The name of the cluster to create. (templated)</li>
<li><strong>project_id</strong> (<em>string</em>) &#x2013; The ID of the google cloud project in which
the cluster runs. (templated)</li>
<li><strong>region</strong> (<em>string</em>) &#x2013; leave as &#x2018;global&#x2019;, might become relevant in the future. (templated)</li>
<li><strong>gcp_conn_id</strong> (<em>string</em>) &#x2013; The connection ID to use connecting to Google Cloud Platform.</li>
<li><strong>delegate_to</strong> (<em>string</em>) &#x2013; The account to impersonate, if any.
For this to work, the service account making the request must have domain-wide
delegation enabled.</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.contrib.operators.dataproc_operator.DataProcPigOperator(query=None, query_uri=None, variables=None, job_name=&apos;{{task.task_id}}_{{ds_nodash}}&apos;, cluster_name=&apos;cluster-1&apos;, dataproc_pig_properties=None, dataproc_pig_jars=None, gcp_conn_id=&apos;google_cloud_default&apos;, delegate_to=None, region=&apos;global&apos;, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Start a Pig query Job on a Cloud DataProc cluster. The parameters of the operation
will be passed to the cluster.</p>
<p>It&#x2019;s a good practice to define dataproc_* parameters in the default_args of the dag
like the cluster name and UDFs.</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">default_args</span> <span class="o">=</span> <span class="p">{</span>
    <span class="s1">&apos;cluster_name&apos;</span><span class="p">:</span> <span class="s1">&apos;cluster-1&apos;</span><span class="p">,</span>
    <span class="s1">&apos;dataproc_pig_jars&apos;</span><span class="p">:</span> <span class="p">[</span>
        <span class="s1">&apos;gs://example/udf/jar/datafu/1.2.0/datafu.jar&apos;</span><span class="p">,</span>
        <span class="s1">&apos;gs://example/udf/jar/gpig/1.2/gpig.jar&apos;</span>
    <span class="p">]</span>
<span class="p">}</span>
</pre>
</div>
</div>
<p>You can pass a pig script as string or file reference. Use variables to pass on
variables for the pig script to be resolved on the cluster or use the parameters to
be resolved in the script as template parameters.</p>
<p><strong>Example</strong>:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">t1</span> <span class="o">=</span> <span class="n">DataProcPigOperator</span><span class="p">(</span>
        <span class="n">task_id</span><span class="o">=</span><span class="s1">&apos;dataproc_pig&apos;</span><span class="p">,</span>
        <span class="n">query</span><span class="o">=</span><span class="s1">&apos;a_pig_script.pig&apos;</span><span class="p">,</span>
        <span class="n">variables</span><span class="o">=</span><span class="p">{</span><span class="s1">&apos;out&apos;</span><span class="p">:</span> <span class="s1">&apos;gs://example/output/{{ds}}&apos;</span><span class="p">},</span>
        <span class="n">dag</span><span class="o">=</span><span class="n">dag</span><span class="p">)</span>
</pre>
</div>
</div>
<div class="admonition seealso">
<p class="first admonition-title">See also</p>
<p class="last">For more detail on about job submission have a look at the reference:
<a class="reference external" href="https://cloud.google.com/dataproc/reference/rest/v1/projects.regions.jobs">https://cloud.google.com/dataproc/reference/rest/v1/projects.regions.jobs</a></p>
</div>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>query</strong> (<em>string</em>) &#x2013; The query or reference to the query
file (pg or pig extension). (templated)</li>
<li><strong>query_uri</strong> (<em>string</em>) &#x2013; The uri of a pig script on Cloud Storage.</li>
<li><strong>variables</strong> (<em>dict</em>) &#x2013; Map of named parameters for the query. (templated)</li>
<li><strong>job_name</strong> (<em>string</em>) &#x2013; The job name used in the DataProc cluster. This
name by default is the task_id appended with the execution data, but can
be templated. The name will always be appended with a random number to
avoid name clashes. (templated)</li>
<li><strong>cluster_name</strong> (<em>string</em>) &#x2013; The name of the DataProc cluster. (templated)</li>
<li><strong>dataproc_pig_properties</strong> (<em>dict</em>) &#x2013; Map for the Pig properties. Ideal to put in
default arguments</li>
<li><strong>dataproc_pig_jars</strong> (<em>list</em>) &#x2013; URIs to jars provisioned in Cloud Storage (example: for
UDFs and libs) and are ideal to put in default arguments.</li>
<li><strong>gcp_conn_id</strong> (<em>string</em>) &#x2013; The connection ID to use connecting to Google Cloud Platform.</li>
<li><strong>delegate_to</strong> (<em>string</em>) &#x2013; The account to impersonate, if any.
For this to work, the service account making the request must have domain-wide
delegation enabled.</li>
<li><strong>region</strong> (<em>string</em>) &#x2013; The specified region where the dataproc cluster is created.</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.contrib.operators.dataproc_operator.DataProcHiveOperator(query=None, query_uri=None, variables=None, job_name=&apos;{{task.task_id}}_{{ds_nodash}}&apos;, cluster_name=&apos;cluster-1&apos;, dataproc_hive_properties=None, dataproc_hive_jars=None, gcp_conn_id=&apos;google_cloud_default&apos;, delegate_to=None, region=&apos;global&apos;, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Start a Hive query Job on a Cloud DataProc cluster.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>query</strong> (<em>string</em>) &#x2013; The query or reference to the query file (q extension).</li>
<li><strong>query_uri</strong> (<em>string</em>) &#x2013; The uri of a hive script on Cloud Storage.</li>
<li><strong>variables</strong> (<em>dict</em>) &#x2013; Map of named parameters for the query.</li>
<li><strong>job_name</strong> (<em>string</em>) &#x2013; The job name used in the DataProc cluster. This name by default
is the task_id appended with the execution data, but can be templated. The
name will always be appended with a random number to avoid name clashes.</li>
<li><strong>cluster_name</strong> (<em>string</em>) &#x2013; The name of the DataProc cluster.</li>
<li><strong>dataproc_hive_properties</strong> (<em>dict</em>) &#x2013; Map for the Pig properties. Ideal to put in
default arguments</li>
<li><strong>dataproc_hive_jars</strong> (<em>list</em>) &#x2013; URIs to jars provisioned in Cloud Storage (example: for
UDFs and libs) and are ideal to put in default arguments.</li>
<li><strong>gcp_conn_id</strong> (<em>string</em>) &#x2013; The connection ID to use connecting to Google Cloud Platform.</li>
<li><strong>delegate_to</strong> (<em>string</em>) &#x2013; The account to impersonate, if any.
For this to work, the service account making the request must have domain-wide
delegation enabled.</li>
<li><strong>region</strong> (<em>string</em>) &#x2013; The specified region where the dataproc cluster is created.</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.contrib.operators.dataproc_operator.DataProcSparkSqlOperator(query=None, query_uri=None, variables=None, job_name=&apos;{{task.task_id}}_{{ds_nodash}}&apos;, cluster_name=&apos;cluster-1&apos;, dataproc_spark_properties=None, dataproc_spark_jars=None, gcp_conn_id=&apos;google_cloud_default&apos;, delegate_to=None, region=&apos;global&apos;, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Start a Spark SQL query Job on a Cloud DataProc cluster.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>query</strong> (<em>string</em>) &#x2013; The query or reference to the query file (q extension). (templated)</li>
<li><strong>query_uri</strong> (<em>string</em>) &#x2013; The uri of a spark sql script on Cloud Storage.</li>
<li><strong>variables</strong> (<em>dict</em>) &#x2013; Map of named parameters for the query. (templated)</li>
<li><strong>job_name</strong> (<em>string</em>) &#x2013; The job name used in the DataProc cluster. This
name by default is the task_id appended with the execution data, but can
be templated. The name will always be appended with a random number to
avoid name clashes. (templated)</li>
<li><strong>cluster_name</strong> (<em>string</em>) &#x2013; The name of the DataProc cluster. (templated)</li>
<li><strong>dataproc_spark_properties</strong> (<em>dict</em>) &#x2013; Map for the Pig properties. Ideal to put in
default arguments</li>
<li><strong>dataproc_spark_jars</strong> (<em>list</em>) &#x2013; URIs to jars provisioned in Cloud Storage (example:
for UDFs and libs) and are ideal to put in default arguments.</li>
<li><strong>gcp_conn_id</strong> (<em>string</em>) &#x2013; The connection ID to use connecting to Google Cloud Platform.</li>
<li><strong>delegate_to</strong> (<em>string</em>) &#x2013; The account to impersonate, if any.
For this to work, the service account making the request must have domain-wide
delegation enabled.</li>
<li><strong>region</strong> (<em>string</em>) &#x2013; The specified region where the dataproc cluster is created.</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.contrib.operators.dataproc_operator.DataProcSparkOperator(main_jar=None, main_class=None, arguments=None, archives=None, files=None, job_name=&apos;{{task.task_id}}_{{ds_nodash}}&apos;, cluster_name=&apos;cluster-1&apos;, dataproc_spark_properties=None, dataproc_spark_jars=None, gcp_conn_id=&apos;google_cloud_default&apos;, delegate_to=None, region=&apos;global&apos;, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Start a Spark Job on a Cloud DataProc cluster.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>main_jar</strong> (<em>string</em>) &#x2013; URI of the job jar provisioned on Cloud Storage. (use this or
the main_class, not both together).</li>
<li><strong>main_class</strong> (<em>string</em>) &#x2013; Name of the job class. (use this or the main_jar, not both
together).</li>
<li><strong>arguments</strong> (<em>list</em>) &#x2013; Arguments for the job. (templated)</li>
<li><strong>archives</strong> (<em>list</em>) &#x2013; List of archived files that will be unpacked in the work
directory. Should be stored in Cloud Storage.</li>
<li><strong>files</strong> (<em>list</em>) &#x2013; List of files to be copied to the working directory</li>
<li><strong>job_name</strong> (<em>string</em>) &#x2013; The job name used in the DataProc cluster. This
name by default is the task_id appended with the execution data, but can
be templated. The name will always be appended with a random number to
avoid name clashes. (templated)</li>
<li><strong>cluster_name</strong> (<em>string</em>) &#x2013; The name of the DataProc cluster. (templated)</li>
<li><strong>dataproc_spark_properties</strong> (<em>dict</em>) &#x2013; Map for the Pig properties. Ideal to put in
default arguments</li>
<li><strong>dataproc_spark_jars</strong> (<em>list</em>) &#x2013; URIs to jars provisioned in Cloud Storage (example:
for UDFs and libs) and are ideal to put in default arguments.</li>
<li><strong>gcp_conn_id</strong> (<em>string</em>) &#x2013; The connection ID to use connecting to Google Cloud Platform.</li>
<li><strong>delegate_to</strong> (<em>string</em>) &#x2013; The account to impersonate, if any.
For this to work, the service account making the request must have domain-wide
delegation enabled.</li>
<li><strong>region</strong> (<em>string</em>) &#x2013; The specified region where the dataproc cluster is created.</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.contrib.operators.dataproc_operator.DataProcHadoopOperator(main_jar=None, main_class=None, arguments=None, archives=None, files=None, job_name=&apos;{{task.task_id}}_{{ds_nodash}}&apos;, cluster_name=&apos;cluster-1&apos;, dataproc_hadoop_properties=None, dataproc_hadoop_jars=None, gcp_conn_id=&apos;google_cloud_default&apos;, delegate_to=None, region=&apos;global&apos;, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Start a Hadoop Job on a Cloud DataProc cluster.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>main_jar</strong> (<em>string</em>) &#x2013; URI of the job jar provisioned on Cloud Storage. (use this or
the main_class, not both together).</li>
<li><strong>main_class</strong> (<em>string</em>) &#x2013; Name of the job class. (use this or the main_jar, not both
together).</li>
<li><strong>arguments</strong> (<em>list</em>) &#x2013; Arguments for the job. (templated)</li>
<li><strong>archives</strong> (<em>list</em>) &#x2013; List of archived files that will be unpacked in the work
directory. Should be stored in Cloud Storage.</li>
<li><strong>files</strong> (<em>list</em>) &#x2013; List of files to be copied to the working directory</li>
<li><strong>job_name</strong> (<em>string</em>) &#x2013; The job name used in the DataProc cluster. This
name by default is the task_id appended with the execution data, but can
be templated. The name will always be appended with a random number to
avoid name clashes. (templated)</li>
<li><strong>cluster_name</strong> (<em>string</em>) &#x2013; The name of the DataProc cluster. (templated)</li>
<li><strong>dataproc_hadoop_properties</strong> (<em>dict</em>) &#x2013; Map for the Pig properties. Ideal to put in
default arguments</li>
<li><strong>dataproc_hadoop_jars</strong> (<em>list</em>) &#x2013; URIs to jars provisioned in Cloud Storage (example:
for UDFs and libs) and are ideal to put in default arguments.</li>
<li><strong>gcp_conn_id</strong> (<em>string</em>) &#x2013; The connection ID to use connecting to Google Cloud Platform.</li>
<li><strong>delegate_to</strong> (<em>string</em>) &#x2013; The account to impersonate, if any.
For this to work, the service account making the request must have domain-wide
delegation enabled.</li>
<li><strong>region</strong> (<em>string</em>) &#x2013; The specified region where the dataproc cluster is created.</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.contrib.operators.dataproc_operator.DataProcPySparkOperator(main, arguments=None, archives=None, pyfiles=None, files=None, job_name=&apos;{{task.task_id}}_{{ds_nodash}}&apos;, cluster_name=&apos;cluster-1&apos;, dataproc_pyspark_properties=None, dataproc_pyspark_jars=None, gcp_conn_id=&apos;google_cloud_default&apos;, delegate_to=None, region=&apos;global&apos;, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Start a PySpark Job on a Cloud DataProc cluster.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>main</strong> (<em>string</em>) &#x2013; [Required] The Hadoop Compatible Filesystem (HCFS) URI of the main
Python file to use as the driver. Must be a .py file.</li>
<li><strong>arguments</strong> (<em>list</em>) &#x2013; Arguments for the job. (templated)</li>
<li><strong>archives</strong> (<em>list</em>) &#x2013; List of archived files that will be unpacked in the work
directory. Should be stored in Cloud Storage.</li>
<li><strong>files</strong> (<em>list</em>) &#x2013; List of files to be copied to the working directory</li>
<li><strong>pyfiles</strong> (<em>list</em>) &#x2013; List of Python files to pass to the PySpark framework.
Supported file types: .py, .egg, and .zip</li>
<li><strong>job_name</strong> (<em>string</em>) &#x2013; The job name used in the DataProc cluster. This
name by default is the task_id appended with the execution data, but can
be templated. The name will always be appended with a random number to
avoid name clashes. (templated)</li>
<li><strong>cluster_name</strong> (<em>string</em>) &#x2013; The name of the DataProc cluster.</li>
<li><strong>dataproc_pyspark_properties</strong> (<em>dict</em>) &#x2013; Map for the Pig properties. Ideal to put in
default arguments</li>
<li><strong>dataproc_pyspark_jars</strong> (<em>list</em>) &#x2013; URIs to jars provisioned in Cloud Storage (example:
for UDFs and libs) and are ideal to put in default arguments.</li>
<li><strong>gcp_conn_id</strong> (<em>string</em>) &#x2013; The connection ID to use connecting to Google Cloud Platform.</li>
<li><strong>delegate_to</strong> (<em>string</em>) &#x2013; The account to impersonate, if any.
For this to work, the service account making the request must have
domain-wide delegation enabled.</li>
<li><strong>region</strong> (<em>string</em>) &#x2013; The specified region where the dataproc cluster is created.</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.contrib.operators.dataproc_operator.DataprocWorkflowTemplateBaseOperator(project_id, region=&apos;global&apos;, gcp_conn_id=&apos;google_cloud_default&apos;, delegate_to=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>




<pre>
class airflow.contrib.operators.dataproc_operator.DataprocWorkflowTemplateInstantiateOperator(template_id, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.contrib.operators.dataproc_operator.DataprocWorkflowTemplateBaseOperator" title="airflow.contrib.operators.dataproc_operator.DataprocWorkflowTemplateBaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.contrib.operators.dataproc_operator.DataprocWorkflowTemplateBaseOperator</span></code></a></p>
<p>Instantiate a WorkflowTemplate on Google Cloud Dataproc. The operator will wait
until the WorkflowTemplate is finished executing.</p>
<div class="admonition seealso">
<p class="first admonition-title">See also</p>
<p class="last">Please refer to:
<a class="reference external" href="https://cloud.google.com/dataproc/docs/reference/rest/v1beta2/projects.regions.workflowTemplates/instantiate">https://cloud.google.com/dataproc/docs/reference/rest/v1beta2/projects.regions.workflowTemplates/instantiate</a></p>
</div>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>template_id</strong> (<em>string</em>) &#x2013; The id of the template. (templated)</li>
<li><strong>project_id</strong> (<em>string</em>) &#x2013; The ID of the google cloud project in which
the template runs</li>
<li><strong>region</strong> (<em>string</em>) &#x2013; leave as &#x2018;global&#x2019;, might become relevant in the future</li>
<li><strong>gcp_conn_id</strong> (<em>string</em>) &#x2013; The connection ID to use connecting to Google Cloud Platform.</li>
<li><strong>delegate_to</strong> (<em>string</em>) &#x2013; The account to impersonate, if any.
For this to work, the service account making the request must have domain-wide
delegation enabled.</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.contrib.operators.dataproc_operator.DataprocWorkflowTemplateInstantiateInlineOperator(template, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.contrib.operators.dataproc_operator.DataprocWorkflowTemplateBaseOperator" title="airflow.contrib.operators.dataproc_operator.DataprocWorkflowTemplateBaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.contrib.operators.dataproc_operator.DataprocWorkflowTemplateBaseOperator</span></code></a></p>
<p>Instantiate a WorkflowTemplate Inline on Google Cloud Dataproc. The operator will
wait until the WorkflowTemplate is finished executing.</p>
<div class="admonition seealso">
<p class="first admonition-title">See also</p>
<p class="last">Please refer to:
<a class="reference external" href="https://cloud.google.com/dataproc/docs/reference/rest/v1beta2/projects.regions.workflowTemplates/instantiateInline">https://cloud.google.com/dataproc/docs/reference/rest/v1beta2/projects.regions.workflowTemplates/instantiateInline</a></p>
</div>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>template</strong> (<em>map</em>) &#x2013; The template contents. (templated)</li>
<li><strong>project_id</strong> (<em>string</em>) &#x2013; The ID of the google cloud project in which
the template runs</li>
<li><strong>region</strong> (<em>string</em>) &#x2013; leave as &#x2018;global&#x2019;, might become relevant in the future</li>
<li><strong>gcp_conn_id</strong> (<em>string</em>) &#x2013; The connection ID to use connecting to Google Cloud Platform.</li>
<li><strong>delegate_to</strong> (<em>string</em>) &#x2013; The account to impersonate, if any.
For this to work, the service account making the request must have domain-wide
delegation enabled.</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.contrib.operators.datastore_export_operator.DatastoreExportOperator(bucket, namespace=None, datastore_conn_id=&apos;google_cloud_default&apos;, cloud_storage_conn_id=&apos;google_cloud_default&apos;, delegate_to=None, entity_filter=None, labels=None, polling_interval_in_seconds=10, overwrite_existing=False, xcom_push=False, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Export entities from Google Cloud Datastore to Cloud Storage</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>bucket</strong> (<em>string</em>) &#x2013; name of the cloud storage bucket to backup data</li>
<li><strong>namespace</strong> (<em>str</em>) &#x2013; optional namespace path in the specified Cloud Storage bucket
to backup data. If this namespace does not exist in GCS, it will be created.</li>
<li><strong>datastore_conn_id</strong> (<em>string</em>) &#x2013; the name of the Datastore connection id to use</li>
<li><strong>cloud_storage_conn_id</strong> (<em>string</em>) &#x2013; the name of the cloud storage connection id to
force-write backup</li>
<li><strong>delegate_to</strong> (<em>string</em>) &#x2013; The account to impersonate, if any.
For this to work, the service account making the request must have domain-wide
delegation enabled.</li>
<li><strong>entity_filter</strong> (<em>dict</em>) &#x2013; description of what data from the project is included in the
export, refer to
<a class="reference external" href="https://cloud.google.com/datastore/docs/reference/rest/Shared.Types/EntityFilter">https://cloud.google.com/datastore/docs/reference/rest/Shared.Types/EntityFilter</a></li>
<li><strong>labels</strong> (<em>dict</em>) &#x2013; client-assigned labels for cloud storage</li>
<li><strong>polling_interval_in_seconds</strong> (<em>int</em>) &#x2013; number of seconds to wait before polling for
execution status again</li>
<li><strong>overwrite_existing</strong> (<em>bool</em>) &#x2013; if the storage bucket + namespace is not empty, it will be
emptied prior to exports. This enables overwriting existing backups.</li>
<li><strong>xcom_push</strong> (<em>bool</em>) &#x2013; push operation name to xcom for reference</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.contrib.operators.datastore_import_operator.DatastoreImportOperator(bucket, file, namespace=None, entity_filter=None, labels=None, datastore_conn_id=&apos;google_cloud_default&apos;, delegate_to=None, polling_interval_in_seconds=10, xcom_push=False, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Import entities from Cloud Storage to Google Cloud Datastore</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>bucket</strong> (<em>string</em>) &#x2013; container in Cloud Storage to store data</li>
<li><strong>file</strong> (<em>string</em>) &#x2013; path of the backup metadata file in the specified Cloud Storage bucket.
It should have the extension .overall_export_metadata</li>
<li><strong>namespace</strong> (<em>str</em>) &#x2013; optional namespace of the backup metadata file in
the specified Cloud Storage bucket.</li>
<li><strong>entity_filter</strong> (<em>dict</em>) &#x2013; description of what data from the project is included in
the export, refer to
<a class="reference external" href="https://cloud.google.com/datastore/docs/reference/rest/Shared.Types/EntityFilter">https://cloud.google.com/datastore/docs/reference/rest/Shared.Types/EntityFilter</a></li>
<li><strong>labels</strong> (<em>dict</em>) &#x2013; client-assigned labels for cloud storage</li>
<li><strong>datastore_conn_id</strong> (<em>string</em>) &#x2013; the name of the connection id to use</li>
<li><strong>delegate_to</strong> (<em>string</em>) &#x2013; The account to impersonate, if any.
For this to work, the service account making the request must have domain-wide
delegation enabled.</li>
<li><strong>polling_interval_in_seconds</strong> (<em>int</em>) &#x2013; number of seconds to wait before polling for
execution status again</li>
<li><strong>xcom_push</strong> (<em>bool</em>) &#x2013; push operation name to xcom for reference</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.contrib.operators.discord_webhook_operator.DiscordWebhookOperator(http_conn_id=None, webhook_endpoint=None, message=&apos;&apos;, username=None, avatar_url=None, tts=False, proxy=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.operators.http_operator.SimpleHttpOperator" title="airflow.operators.http_operator.SimpleHttpOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.operators.http_operator.SimpleHttpOperator</span></code></a></p>
<p>This operator allows you to post messages to Discord using incoming webhooks.
Takes a Discord connection ID with a default relative webhook endpoint. The
default endpoint can be overridden using the webhook_endpoint parameter
(<a class="reference external" href="https://discordapp.com/developers/docs/resources/webhook">https://discordapp.com/developers/docs/resources/webhook</a>).</p>
<p>Each Discord webhook can be pre-configured to use a specific username and
avatar_url. You can override these defaults in this operator.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>http_conn_id</strong> (<em>str</em>) &#x2013; Http connection ID with host as &#x201C;<a class="reference external" href="https://discord.com/api/">https://discord.com/api/</a>&#x201D; and
default webhook endpoint in the extra field in the form of
{&#x201C;webhook_endpoint&#x201D;: &#x201C;webhooks/{webhook.id}/{webhook.token}&#x201D;}</li>
<li><strong>webhook_endpoint</strong> (<em>str</em>) &#x2013; Discord webhook endpoint in the form of
&#x201C;webhooks/{webhook.id}/{webhook.token}&#x201D;</li>
<li><strong>message</strong> (<em>str</em>) &#x2013; The message you want to send to your Discord channel
(max 2000 characters). (templated)</li>
<li><strong>username</strong> (<em>str</em>) &#x2013; Override the default username of the webhook. (templated)</li>
<li><strong>avatar_url</strong> (<em>str</em>) &#x2013; Override the default avatar of the webhook</li>
<li><strong>tts</strong> (<em>bool</em>) &#x2013; Is a text-to-speech message</li>
<li><strong>proxy</strong> (<em>str</em>) &#x2013; Proxy to use to make the Discord webhook call</li>
</ul>
</td>
</tr>
</tbody>
</table>

<pre>
execute(context)</pre>
<p>Call the DiscordWebhookHook to post message</p>







<pre>
class airflow.contrib.operators.druid_operator.DruidOperator(json_index_file, druid_ingest_conn_id=&apos;druid_ingest_default&apos;, max_ingestion_time=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Allows to submit a task directly to druid</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>json_index_file</strong> (<em>str</em>) &#x2013; The filepath to the druid index specification</li>
<li><strong>druid_ingest_conn_id</strong> (<em>str</em>) &#x2013; The connection id of the Druid overlord which
accepts index jobs</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.contrib.operators.ecs_operator.ECSOperator(task_definition, cluster, overrides, aws_conn_id=None, region_name=None, launch_type=&apos;EC2&apos;, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Execute a task on AWS EC2 Container Service</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first simple">
<li><strong>task_definition</strong> (<em>str</em>) &#x2013; the task definition name on EC2 Container Service</li>
<li><strong>cluster</strong> (<em>str</em>) &#x2013; the cluster name on EC2 Container Service</li>
<li><strong>aws_conn_id</strong> (<em>str</em>) &#x2013; connection id of AWS credentials / region name. If None,
credential boto3 strategy will be used
(<a class="reference external" href="http://boto3.readthedocs.io/en/latest/guide/configuration.html">http://boto3.readthedocs.io/en/latest/guide/configuration.html</a>).</li>
<li><strong>region_name</strong> &#x2013; region name to use in AWS Hook.
Override the region_name in connection (if provided)</li>
<li><strong>launch_type</strong> &#x2013; the launch type on which to run your task (&#x2018;EC2&#x2019; or &#x2018;FARGATE&#x2019;)</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Param:</th>
<td class="field-body"><p class="first">overrides: the same parameter that boto3 will receive (templated):
<a class="reference external" href="http://boto3.readthedocs.org/en/latest/reference/services/ecs.html#ECS.Client.run_task">http://boto3.readthedocs.org/en/latest/reference/services/ecs.html#ECS.Client.run_task</a></p>
</td>
</tr>
<tr class="field-odd field"><th class="field-name">Type:</th>
<td class="field-body"><p class="first">overrides: dict</p>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Type:</th>
<td class="field-body"><p class="first last">launch_type: str</p>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.contrib.operators.emr_add_steps_operator.EmrAddStepsOperator(job_flow_id, aws_conn_id=&apos;s3_default&apos;, steps=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>An operator that adds steps to an existing EMR job_flow.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>job_flow_id</strong> &#x2013; id of the JobFlow to add steps to. (templated)</li>
<li><strong>aws_conn_id</strong> (<em>str</em>) &#x2013; aws connection to uses</li>
<li><strong>steps</strong> (<em>list</em>) &#x2013; boto3 style steps to be added to the jobflow. (templated)</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.contrib.operators.emr_create_job_flow_operator.EmrCreateJobFlowOperator(aws_conn_id=&apos;s3_default&apos;, emr_conn_id=&apos;emr_default&apos;, job_flow_overrides=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Creates an EMR JobFlow, reading the config from the EMR connection.
A dictionary of JobFlow overrides can be passed that override
the config from the connection.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>aws_conn_id</strong> (<em>str</em>) &#x2013; aws connection to uses</li>
<li><strong>emr_conn_id</strong> (<em>str</em>) &#x2013; emr connection to use</li>
<li><strong>job_flow_overrides</strong> &#x2013; boto3 style arguments to override
emr_connection extra. (templated)</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.contrib.operators.emr_terminate_job_flow_operator.EmrTerminateJobFlowOperator(job_flow_id, aws_conn_id=&apos;s3_default&apos;, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Operator to terminate EMR JobFlows.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>job_flow_id</strong> &#x2013; id of the JobFlow to terminate. (templated)</li>
<li><strong>aws_conn_id</strong> (<em>str</em>) &#x2013; aws connection to uses</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.contrib.operators.file_to_gcs.FileToGoogleCloudStorageOperator(src, dst, bucket, google_cloud_storage_conn_id=&apos;google_cloud_default&apos;, mime_type=&apos;application/octet-stream&apos;, delegate_to=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Uploads a file to Google Cloud Storage</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>src</strong> (<em>string</em>) &#x2013; Path to the local file. (templated)</li>
<li><strong>dst</strong> (<em>string</em>) &#x2013; Destination path within the specified bucket. (templated)</li>
<li><strong>bucket</strong> (<em>string</em>) &#x2013; The bucket to upload to. (templated)</li>
<li><strong>google_cloud_storage_conn_id</strong> (<em>string</em>) &#x2013; The Airflow connection ID to upload with</li>
<li><strong>mime_type</strong> (<em>string</em>) &#x2013; The mime-type string</li>
<li><strong>delegate_to</strong> (<em>string</em>) &#x2013; The account to impersonate, if any</li>
</ul>
</td>
</tr>
</tbody>
</table>

<pre>
execute(context)</pre>
<p>Uploads the file to Google cloud storage</p>







<pre>
class airflow.contrib.operators.file_to_wasb.FileToWasbOperator(file_path, container_name, blob_name, wasb_conn_id=&apos;wasb_default&apos;, load_options=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Uploads a file to Azure Blob Storage.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>file_path</strong> (<em>str</em>) &#x2013; Path to the file to load. (templated)</li>
<li><strong>container_name</strong> (<em>str</em>) &#x2013; Name of the container. (templated)</li>
<li><strong>blob_name</strong> (<em>str</em>) &#x2013; Name of the blob. (templated)</li>
<li><strong>wasb_conn_id</strong> (<em>str</em>) &#x2013; Reference to the wasb connection.</li>
<li><strong>load_options</strong> (<em>dict</em>) &#x2013; Optional keyword arguments that
<cite>WasbHook.load_file()</cite> takes.</li>
</ul>
</td>
</tr>
</tbody>
</table>

<pre>
execute(context)</pre>
<p>Upload a file to Azure Blob Storage.</p>







<pre>
class airflow.contrib.operators.gcp_container_operator.GKEClusterCreateOperator(project_id, location, body={}, gcp_conn_id=&apos;google_cloud_default&apos;, api_version=&apos;v2&apos;, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>




<pre>
class airflow.contrib.operators.gcp_container_operator.GKEClusterDeleteOperator(project_id, name, location, gcp_conn_id=&apos;google_cloud_default&apos;, api_version=&apos;v2&apos;, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>




<pre>
class airflow.contrib.operators.gcs_download_operator.GoogleCloudStorageDownloadOperator(bucket, object, filename=None, store_to_xcom_key=None, google_cloud_storage_conn_id=&apos;google_cloud_default&apos;, delegate_to=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Downloads a file from Google Cloud Storage.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>bucket</strong> (<em>string</em>) &#x2013; The Google cloud storage bucket where the object is. (templated)</li>
<li><strong>object</strong> (<em>string</em>) &#x2013; The name of the object to download in the Google cloud
storage bucket. (templated)</li>
<li><strong>filename</strong> (<em>string</em>) &#x2013; The file path on the local file system (where the
operator is being executed) that the file should be downloaded to. (templated)
If no filename passed, the downloaded data will not be stored on the local file
system.</li>
<li><strong>store_to_xcom_key</strong> (<em>string</em>) &#x2013; If this param is set, the operator will push
the contents of the downloaded file to XCom with the key set in this
parameter. If not set, the downloaded data will not be pushed to XCom. (templated)</li>
<li><strong>google_cloud_storage_conn_id</strong> (<em>string</em>) &#x2013; The connection ID to use when
connecting to Google cloud storage.</li>
<li><strong>delegate_to</strong> (<em>string</em>) &#x2013; The account to impersonate, if any.
For this to work, the service account making the request must have
domain-wide delegation enabled.</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.contrib.operators.gcs_list_operator.GoogleCloudStorageListOperator(bucket, prefix=None, delimiter=None, google_cloud_storage_conn_id=&apos;google_cloud_default&apos;, delegate_to=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>List all objects from the bucket with the give string prefix and delimiter in name.</p>

<pre>This operator returns a python list with the name of objects which can be used by</pre>
<cite>xcom</cite> in the downstream task.

<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>bucket</strong> (<em>string</em>) &#x2013; The Google cloud storage bucket to find the objects. (templated)</li>
<li><strong>prefix</strong> (<em>string</em>) &#x2013; Prefix string which filters objects whose name begin with
this prefix. (templated)</li>
<li><strong>delimiter</strong> (<em>string</em>) &#x2013; The delimiter by which you want to filter the objects. (templated)
For e.g to lists the CSV files from in a directory in GCS you would use
delimiter=&#x2019;.csv&#x2019;.</li>
<li><strong>google_cloud_storage_conn_id</strong> (<em>string</em>) &#x2013; The connection ID to use when
connecting to Google cloud storage.</li>
<li><strong>delegate_to</strong> (<em>string</em>) &#x2013; The account to impersonate, if any.
For this to work, the service account making the request must have
domain-wide delegation enabled.</li>
</ul>
</td>
</tr>
</tbody>
</table>

<pre>Example:</pre>
<p class="first">The following Operator would list all the Avro files from <code class="docutils literal notranslate"><span class="pre">sales/sales-2017</span></code>
folder in <code class="docutils literal notranslate"><span class="pre">data</span></code> bucket.</p>
<div class="last highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">GCS_Files</span> <span class="o">=</span> <span class="n">GoogleCloudStorageListOperator</span><span class="p">(</span>
    <span class="n">task_id</span><span class="o">=</span><span class="s1">&apos;GCS_Files&apos;</span><span class="p">,</span>
    <span class="n">bucket</span><span class="o">=</span><span class="s1">&apos;data&apos;</span><span class="p">,</span>
    <span class="n">prefix</span><span class="o">=</span><span class="s1">&apos;sales/sales-2017/&apos;</span><span class="p">,</span>
    <span class="n">delimiter</span><span class="o">=</span><span class="s1">&apos;.avro&apos;</span><span class="p">,</span>
    <span class="n">google_cloud_storage_conn_id</span><span class="o">=</span><span class="n">google_cloud_conn_id</span>
<span class="p">)</span>
</pre>
</div>
</div>






<pre>
class airflow.contrib.operators.gcs_operator.GoogleCloudStorageCreateBucketOperator(bucket_name, storage_class=&apos;MULTI_REGIONAL&apos;, location=&apos;US&apos;, project_id=None, labels=None, google_cloud_storage_conn_id=&apos;google_cloud_default&apos;, delegate_to=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Creates a new bucket. Google Cloud Storage uses a flat namespace,
so you can&#x2019;t create a bucket with a name that is already in use.</p>
<blockquote>
<div><div class="admonition seealso">
<p class="first admonition-title">See also</p>
<p class="last">For more information, see Bucket Naming Guidelines:
<a class="reference external" href="https://cloud.google.com/storage/docs/bucketnaming.html#requirements">https://cloud.google.com/storage/docs/bucketnaming.html#requirements</a></p>
</div>
</div>
</blockquote>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>bucket_name</strong> (<em>string</em>) &#x2013; The name of the bucket. (templated)</li>
<li><strong>storage_class</strong> (<em>string</em>) &#x2013; <p>This defines how objects in the bucket are stored
and determines the SLA and the cost of storage (templated). Values include</p>
<ul>
<li><code class="docutils literal notranslate"><span class="pre">MULTI_REGIONAL</span></code></li>
<li><code class="docutils literal notranslate"><span class="pre">REGIONAL</span></code></li>
<li><code class="docutils literal notranslate"><span class="pre">STANDARD</span></code></li>
<li><code class="docutils literal notranslate"><span class="pre">NEARLINE</span></code></li>
<li><code class="docutils literal notranslate"><span class="pre">COLDLINE</span></code>.</li>
</ul>
<p>If this value is not specified when the bucket is
created, it will default to STANDARD.</p>
</li>
<li><strong>location</strong> (<em>string</em>) &#x2013; <p>The location of the bucket. (templated)
Object data for objects in the bucket resides in physical storage
within this region. Defaults to US.</p>
<div class="admonition seealso">
<p class="first admonition-title">See also</p>
<p class="last"><a class="reference external" href="https://developers.google.com/storage/docs/bucket-locations">https://developers.google.com/storage/docs/bucket-locations</a></p>
</div>
</li>
<li><strong>project_id</strong> (<em>string</em>) &#x2013; The ID of the GCP Project. (templated)</li>
<li><strong>labels</strong> (<em>dict</em>) &#x2013; User-provided labels, in key/value pairs.</li>
<li><strong>google_cloud_storage_conn_id</strong> (<em>string</em>) &#x2013; The connection ID to use when
connecting to Google cloud storage.</li>
<li><strong>delegate_to</strong> (<em>string</em>) &#x2013; The account to impersonate, if any.
For this to work, the service account making the request must
have domain-wide delegation enabled.</li>
</ul>
</td>
</tr>
</tbody>
</table>

<pre>Example:</pre>
<p class="first">The following Operator would create a new bucket <code class="docutils literal notranslate"><span class="pre">test-bucket</span></code>
with <code class="docutils literal notranslate"><span class="pre">MULTI_REGIONAL</span></code> storage class in <code class="docutils literal notranslate"><span class="pre">EU</span></code> region</p>
<div class="last highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">CreateBucket</span> <span class="o">=</span> <span class="n">GoogleCloudStorageCreateBucketOperator</span><span class="p">(</span>
    <span class="n">task_id</span><span class="o">=</span><span class="s1">&apos;CreateNewBucket&apos;</span><span class="p">,</span>
    <span class="n">bucket_name</span><span class="o">=</span><span class="s1">&apos;test-bucket&apos;</span><span class="p">,</span>
    <span class="n">storage_class</span><span class="o">=</span><span class="s1">&apos;MULTI_REGIONAL&apos;</span><span class="p">,</span>
    <span class="n">location</span><span class="o">=</span><span class="s1">&apos;EU&apos;</span><span class="p">,</span>
    <span class="n">labels</span><span class="o">=</span><span class="p">{</span><span class="s1">&apos;env&apos;</span><span class="p">:</span> <span class="s1">&apos;dev&apos;</span><span class="p">,</span> <span class="s1">&apos;team&apos;</span><span class="p">:</span> <span class="s1">&apos;airflow&apos;</span><span class="p">},</span>
    <span class="n">google_cloud_storage_conn_id</span><span class="o">=</span><span class="s1">&apos;airflow-service-account&apos;</span>
<span class="p">)</span>
</pre>
</div>
</div>






<pre>
class airflow.contrib.operators.gcs_to_bq.GoogleCloudStorageToBigQueryOperator(bucket, source_objects, destination_project_dataset_table, schema_fields=None, schema_object=None, source_format=&apos;CSV&apos;, compression=&apos;NONE&apos;, create_disposition=&apos;CREATE_IF_NEEDED&apos;, skip_leading_rows=0, write_disposition=&apos;WRITE_EMPTY&apos;, field_delimiter=&apos;, &apos;, max_bad_records=0, quote_character=None, ignore_unknown_values=False, allow_quoted_newlines=False, allow_jagged_rows=False, max_id_key=None, bigquery_conn_id=&apos;bigquery_default&apos;, google_cloud_storage_conn_id=&apos;google_cloud_default&apos;, delegate_to=None, schema_update_options=(), src_fmt_configs={}, external_table=False, time_partitioning={}, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Loads files from Google cloud storage into BigQuery.</p>
<p>The schema to be used for the BigQuery table may be specified in one of
two ways. You may either directly pass the schema fields in, or you may
point the operator to a Google cloud storage object name. The object in
Google cloud storage must be a JSON file with the schema fields in it.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>bucket</strong> (<em>string</em>) &#x2013; The bucket to load from. (templated)</li>
<li><strong>source_objects</strong> &#x2013; List of Google cloud storage URIs to load from. (templated)
If source_format is &#x2018;DATASTORE_BACKUP&#x2019;, the list must only contain a single URI.</li>
<li><strong>destination_project_dataset_table</strong> (<em>string</em>) &#x2013; The dotted (&lt;project&gt;.)&lt;dataset&gt;.&lt;table&gt;
BigQuery table to load data into. If &lt;project&gt; is not included,
project will be the project defined in the connection json. (templated)</li>
<li><strong>schema_fields</strong> (<em>list</em>) &#x2013; If set, the schema field list as defined here:
<a class="reference external" href="https://cloud.google.com/bigquery/docs/reference/v2/jobs#configuration.load">https://cloud.google.com/bigquery/docs/reference/v2/jobs#configuration.load</a>
Should not be set when source_format is &#x2018;DATASTORE_BACKUP&#x2019;.</li>
<li><strong>schema_object</strong> &#x2013; If set, a GCS object path pointing to a .json file that
contains the schema for the table. (templated)</li>
<li><strong>schema_object</strong> &#x2013; string</li>
<li><strong>source_format</strong> (<em>string</em>) &#x2013; File format to export.</li>
<li><strong>compression</strong> (<em>string</em>) &#x2013; [Optional] The compression type of the data source.
Possible values include GZIP and NONE.
The default value is NONE.
This setting is ignored for Google Cloud Bigtable,
Google Cloud Datastore backups and Avro formats.</li>
<li><strong>create_disposition</strong> (<em>string</em>) &#x2013; The create disposition if the table doesn&#x2019;t exist.</li>
<li><strong>skip_leading_rows</strong> (<em>int</em>) &#x2013; Number of rows to skip when loading from a CSV.</li>
<li><strong>write_disposition</strong> (<em>string</em>) &#x2013; The write disposition if the table already exists.</li>
<li><strong>field_delimiter</strong> (<em>string</em>) &#x2013; The delimiter to use when loading from a CSV.</li>
<li><strong>max_bad_records</strong> (<em>int</em>) &#x2013; The maximum number of bad records that BigQuery can
ignore when running the job.</li>
<li><strong>quote_character</strong> (<em>string</em>) &#x2013; The value that is used to quote data sections in a CSV file.</li>
<li><strong>ignore_unknown_values</strong> (<em>bool</em>) &#x2013; [Optional] Indicates if BigQuery should allow
extra values that are not represented in the table schema.
If true, the extra values are ignored. If false, records with extra columns
are treated as bad records, and if there are too many bad records, an
invalid error is returned in the job result.</li>
<li><strong>allow_quoted_newlines</strong> (<em>boolean</em>) &#x2013; Whether to allow quoted newlines (true) or not (false).</li>
<li><strong>allow_jagged_rows</strong> (<em>bool</em>) &#x2013; Accept rows that are missing trailing optional columns.
The missing values are treated as nulls. If false, records with missing trailing
columns are treated as bad records, and if there are too many bad records, an
invalid error is returned in the job result. Only applicable to CSV, ignored
for other formats.</li>
<li><strong>max_id_key</strong> (<em>string</em>) &#x2013; If set, the name of a column in the BigQuery table
that&#x2019;s to be loaded. Thsi will be used to select the MAX value from
BigQuery after the load occurs. The results will be returned by the
execute() command, which in turn gets stored in XCom for future
operators to use. This can be helpful with incremental loads&#x2013;during
future executions, you can pick up from the max ID.</li>
<li><strong>bigquery_conn_id</strong> (<em>string</em>) &#x2013; Reference to a specific BigQuery hook.</li>
<li><strong>google_cloud_storage_conn_id</strong> (<em>string</em>) &#x2013; Reference to a specific Google
cloud storage hook.</li>
<li><strong>delegate_to</strong> (<em>string</em>) &#x2013; The account to impersonate, if any. For this to
work, the service account making the request must have domain-wide
delegation enabled.</li>
<li><strong>schema_update_options</strong> (<em>list</em>) &#x2013; Allows the schema of the destination
table to be updated as a side effect of the load job.</li>
<li><strong>src_fmt_configs</strong> (<em>dict</em>) &#x2013; configure optional fields specific to the source format</li>
<li><strong>external_table</strong> (<em>bool</em>) &#x2013; Flag to specify if the destination table should be
a BigQuery external table. Default Value is False.</li>
<li><strong>time_partitioning</strong> (<em>dict</em>) &#x2013; configure optional time partitioning fields i.e.
partition by field, type and  expiration as per API specifications.
Note that &#x2018;field&#x2019; is not available in concurrency with
dataset.table$partition.</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.contrib.operators.gcs_to_gcs.GoogleCloudStorageToGoogleCloudStorageOperator(source_bucket, source_object, destination_bucket=None, destination_object=None, move_object=False, google_cloud_storage_conn_id=&apos;google_cloud_default&apos;, delegate_to=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Copies objects from a bucket to another, with renaming if requested.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>source_bucket</strong> (<em>string</em>) &#x2013; The source Google cloud storage bucket where the
object is. (templated)</li>
<li><strong>source_object</strong> (<em>string</em>) &#x2013; <p>The source name of the object to copy in the Google cloud
storage bucket. (templated)
If wildcards are used in this argument:</p>
<blockquote>
<div>You can use only one wildcard for objects (filenames) within your
bucket. The wildcard can appear inside the object name or at the
end of the object name. Appending a wildcard to the bucket name is
unsupported.</div>
</blockquote>
</li>
<li><strong>destination_bucket</strong> &#x2013; The destination Google cloud storage bucket</li>
</ul>
</td>
</tr>
</tbody>
</table>
<p>where the object should be. (templated)
:type destination_bucket: string
:param destination_object: The destination name of the object in the</p>
<blockquote>
<div>destination Google cloud storage bucket. (templated)
If a wildcard is supplied in the source_object argument, this is the
prefix that will be prepended to the final destination objects&#x2019; paths.
Note that the source path&#x2019;s part before the wildcard will be removed;
if it needs to be retained it should be appended to destination_object.
For example, with prefix <code class="docutils literal notranslate"><span class="pre">foo/*</span></code> and destination_object <cite>&#x2018;blah/`</cite>, the
file <code class="docutils literal notranslate"><span class="pre">foo/baz</span></code> will be copied to <code class="docutils literal notranslate"><span class="pre">blah/baz</span></code>; to retain the prefix write
the destination_object as e.g. <code class="docutils literal notranslate"><span class="pre">blah/foo</span></code>, in which case the copied file
will be named <code class="docutils literal notranslate"><span class="pre">blah/foo/baz</span></code>.</div>
</blockquote>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><strong>move_object</strong> &#x2013; When move object is True, the object is moved instead</td>
</tr>
</tbody>
</table>

<pre>of copied to the new location.</pre>
This is the equivalent of a mv command as opposed to a
cp command.

<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>google_cloud_storage_conn_id</strong> (<em>string</em>) &#x2013; The connection ID to use when
connecting to Google cloud storage.</li>
<li><strong>delegate_to</strong> (<em>string</em>) &#x2013; The account to impersonate, if any.
For this to work, the service account making the request must have
domain-wide delegation enabled.</li>
</ul>
</td>
</tr>
</tbody>
</table>

<pre>Examples:</pre>
<p class="first">The following Operator would copy a single file named
<code class="docutils literal notranslate"><span class="pre">sales/sales-2017/january.avro</span></code> in the <code class="docutils literal notranslate"><span class="pre">data</span></code> bucket to the file named
<code class="docutils literal notranslate"><span class="pre">copied_sales/2017/january-backup.avro`</span> <span class="pre">in</span> <span class="pre">the</span> <span class="pre">``data_backup</span></code> bucket</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">copy_single_file</span> <span class="o">=</span> <span class="n">GoogleCloudStorageToGoogleCloudStorageOperator</span><span class="p">(</span>
    <span class="n">task_id</span><span class="o">=</span><span class="s1">&apos;copy_single_file&apos;</span><span class="p">,</span>
    <span class="n">source_bucket</span><span class="o">=</span><span class="s1">&apos;data&apos;</span><span class="p">,</span>
    <span class="n">source_object</span><span class="o">=</span><span class="s1">&apos;sales/sales-2017/january.avro&apos;</span><span class="p">,</span>
    <span class="n">destination_bucket</span><span class="o">=</span><span class="s1">&apos;data_backup&apos;</span><span class="p">,</span>
    <span class="n">destination_object</span><span class="o">=</span><span class="s1">&apos;copied_sales/2017/january-backup.avro&apos;</span><span class="p">,</span>
    <span class="n">google_cloud_storage_conn_id</span><span class="o">=</span><span class="n">google_cloud_conn_id</span>
<span class="p">)</span>
</pre>
</div>
</div>
<p>The following Operator would copy all the Avro files from <code class="docutils literal notranslate"><span class="pre">sales/sales-2017</span></code>
folder (i.e. with names starting with that prefix) in <code class="docutils literal notranslate"><span class="pre">data</span></code> bucket to the
<code class="docutils literal notranslate"><span class="pre">copied_sales/2017</span></code> folder in the <code class="docutils literal notranslate"><span class="pre">data_backup</span></code> bucket.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">copy_files</span> <span class="o">=</span> <span class="n">GoogleCloudStorageToGoogleCloudStorageOperator</span><span class="p">(</span>
    <span class="n">task_id</span><span class="o">=</span><span class="s1">&apos;copy_files&apos;</span><span class="p">,</span>
    <span class="n">source_bucket</span><span class="o">=</span><span class="s1">&apos;data&apos;</span><span class="p">,</span>
    <span class="n">source_object</span><span class="o">=</span><span class="s1">&apos;sales/sales-2017/*.avro&apos;</span><span class="p">,</span>
    <span class="n">destination_bucket</span><span class="o">=</span><span class="s1">&apos;data_backup&apos;</span><span class="p">,</span>
    <span class="n">destination_object</span><span class="o">=</span><span class="s1">&apos;copied_sales/2017/&apos;</span><span class="p">,</span>
    <span class="n">google_cloud_storage_conn_id</span><span class="o">=</span><span class="n">google_cloud_conn_id</span>
<span class="p">)</span>
</pre>
</div>
</div>
<p>The following Operator would move all the Avro files from <code class="docutils literal notranslate"><span class="pre">sales/sales-2017</span></code>
folder (i.e. with names starting with that prefix) in <code class="docutils literal notranslate"><span class="pre">data</span></code> bucket to the
same folder in the <code class="docutils literal notranslate"><span class="pre">data_backup</span></code> bucket, deleting the original files in the
process.</p>
<div class="last highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">move_files</span> <span class="o">=</span> <span class="n">GoogleCloudStorageToGoogleCloudStorageOperator</span><span class="p">(</span>
    <span class="n">task_id</span><span class="o">=</span><span class="s1">&apos;move_files&apos;</span><span class="p">,</span>
    <span class="n">source_bucket</span><span class="o">=</span><span class="s1">&apos;data&apos;</span><span class="p">,</span>
    <span class="n">source_object</span><span class="o">=</span><span class="s1">&apos;sales/sales-2017/*.avro&apos;</span><span class="p">,</span>
    <span class="n">destination_bucket</span><span class="o">=</span><span class="s1">&apos;data_backup&apos;</span><span class="p">,</span>
    <span class="n">move_object</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span>
    <span class="n">google_cloud_storage_conn_id</span><span class="o">=</span><span class="n">google_cloud_conn_id</span>
<span class="p">)</span>
</pre>
</div>
</div>






<pre>
class airflow.contrib.operators.gcs_to_s3.GoogleCloudStorageToS3Operator(bucket, prefix=None, delimiter=None, google_cloud_storage_conn_id=&apos;google_cloud_storage_default&apos;, delegate_to=None, dest_aws_conn_id=None, dest_s3_key=None, replace=False, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="integration.html#airflow.contrib.operators.gcs_list_operator.GoogleCloudStorageListOperator" title="airflow.contrib.operators.gcs_list_operator.GoogleCloudStorageListOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.contrib.operators.gcs_list_operator.GoogleCloudStorageListOperator</span></code></a></p>
<p>Synchronizes a Google Cloud Storage bucket with an S3 bucket.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>bucket</strong> (<em>string</em>) &#x2013; The Google Cloud Storage bucket to find the objects. (templated)</li>
<li><strong>prefix</strong> (<em>string</em>) &#x2013; Prefix string which filters objects whose name begin with
this prefix. (templated)</li>
<li><strong>delimiter</strong> (<em>string</em>) &#x2013; The delimiter by which you want to filter the objects. (templated)
For e.g to lists the CSV files from in a directory in GCS you would use
delimiter=&#x2019;.csv&#x2019;.</li>
<li><strong>google_cloud_storage_conn_id</strong> (<em>string</em>) &#x2013; The connection ID to use when
connecting to Google Cloud Storage.</li>
<li><strong>delegate_to</strong> (<em>string</em>) &#x2013; The account to impersonate, if any.
For this to work, the service account making the request must have
domain-wide delegation enabled.</li>
<li><strong>dest_aws_conn_id</strong> (<em>str</em>) &#x2013; The destination S3 connection</li>
<li><strong>dest_s3_key</strong> (<em>str</em>) &#x2013; The base S3 key to be used to store the files. (templated)</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.contrib.operators.hipchat_operator.HipChatAPIOperator(token, base_url=&apos;https://api.hipchat.com/v2&apos;, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Base HipChat Operator.
All derived HipChat operators reference from HipChat&#x2019;s official REST API documentation
at <a class="reference external" href="https://www.hipchat.com/docs/apiv2">https://www.hipchat.com/docs/apiv2</a>. Before using any HipChat API operators you need
to get an authentication token at <a class="reference external" href="https://www.hipchat.com/docs/apiv2/auth">https://www.hipchat.com/docs/apiv2/auth</a>.
In the future additional HipChat operators will be derived from this class as well.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>token</strong> (<em>str</em>) &#x2013; HipChat REST API authentication token</li>
<li><strong>base_url</strong> (<em>str</em>) &#x2013; HipChat REST API base url.</li>
</ul>
</td>
</tr>
</tbody>
</table>

<pre>
prepare_request()</pre>
<p>Used by the execute function. Set the request method, url, and body of HipChat&#x2019;s
REST API call.
Override in child class. Each HipChatAPI child operator is responsible for having
a prepare_request method call which sets self.method, self.url, and self.body.</p>







<pre>
class airflow.contrib.operators.hipchat_operator.HipChatAPISendRoomNotificationOperator(room_id, message, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.contrib.operators.hipchat_operator.HipChatAPIOperator" title="airflow.contrib.operators.hipchat_operator.HipChatAPIOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.contrib.operators.hipchat_operator.HipChatAPIOperator</span></code></a></p>
<p>Send notification to a specific HipChat room.
More info: <a class="reference external" href="https://www.hipchat.com/docs/apiv2/method/send_room_notification">https://www.hipchat.com/docs/apiv2/method/send_room_notification</a></p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>room_id</strong> (<em>str</em>) &#x2013; Room in which to send notification on HipChat. (templated)</li>
<li><strong>message</strong> (<em>str</em>) &#x2013; The message body. (templated)</li>
<li><strong>frm</strong> (<em>str</em>) &#x2013; Label to be shown in addition to sender&#x2019;s name</li>
<li><strong>message_format</strong> (<em>str</em>) &#x2013; How the notification is rendered: html or text</li>
<li><strong>color</strong> (<em>str</em>) &#x2013; Background color of the msg: yellow, green, red, purple, gray, or random</li>
<li><strong>attach_to</strong> (<em>str</em>) &#x2013; The message id to attach this notification to</li>
<li><strong>notify</strong> (<em>bool</em>) &#x2013; Whether this message should trigger a user notification</li>
<li><strong>card</strong> (<em>dict</em>) &#x2013; HipChat-defined card object</li>
</ul>
</td>
</tr>
</tbody>
</table>

<pre>
prepare_request()</pre>
<p>Used by the execute function. Set the request method, url, and body of HipChat&#x2019;s
REST API call.
Override in child class. Each HipChatAPI child operator is responsible for having
a prepare_request method call which sets self.method, self.url, and self.body.</p>







<pre>
class airflow.contrib.operators.hive_to_dynamodb.HiveToDynamoDBTransferOperator(sql, table_name, table_keys, pre_process=None, pre_process_args=None, pre_process_kwargs=None, region_name=None, schema=&apos;default&apos;, hiveserver2_conn_id=&apos;hiveserver2_default&apos;, aws_conn_id=&apos;aws_default&apos;, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Moves data from Hive to DynamoDB, note that for now the data is loaded
into memory before being pushed to DynamoDB, so this operator should
be used for smallish amount of data.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>sql</strong> (<em>str</em>) &#x2013; SQL query to execute against the hive database. (templated)</li>
<li><strong>table_name</strong> (<em>str</em>) &#x2013; target DynamoDB table</li>
<li><strong>table_keys</strong> (<em>list</em>) &#x2013; partition key and sort key</li>
<li><strong>pre_process</strong> (<em>function</em>) &#x2013; implement pre-processing of source data</li>
<li><strong>pre_process_args</strong> (<em>list</em>) &#x2013; list of pre_process function arguments</li>
<li><strong>pre_process_kwargs</strong> (<em>dict</em>) &#x2013; dict of pre_process function arguments</li>
<li><strong>region_name</strong> (<em>str</em>) &#x2013; aws region name (example: us-east-1)</li>
<li><strong>schema</strong> (<em>str</em>) &#x2013; hive database schema</li>
<li><strong>hiveserver2_conn_id</strong> (<em>str</em>) &#x2013; source hive connection</li>
<li><strong>aws_conn_id</strong> (<em>str</em>) &#x2013; aws connection</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.contrib.operators.jenkins_job_trigger_operator.JenkinsJobTriggerOperator(jenkins_connection_id, job_name, parameters=&apos;&apos;, sleep_time=10, max_try_before_job_appears=10, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Trigger a Jenkins Job and monitor it&#x2019;s execution.
This operator depend on python-jenkins library,
version &gt;= 0.4.15 to communicate with jenkins server.
You&#x2019;ll also need to configure a Jenkins connection in the connections screen.
:param jenkins_connection_id: The jenkins connection to use for this job
:type jenkins_connection_id: string
:param job_name: The name of the job to trigger
:type job_name: string
:param parameters: The parameters block to provide to jenkins. (templated)
:type parameters: string
:param sleep_time: How long will the operator sleep between each status
request for the job (min 1, default 10)
:type sleep_time: int
:param max_try_before_job_appears: The maximum number of requests to make</p>
<blockquote>
<div>while waiting for the job to appears on jenkins server (default 10)</div>
</blockquote>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
</tbody>
</table>

<pre>
build_job(jenkins_server)</pre>
<p>This function makes an API call to Jenkins to trigger a build for &#x2018;job_name&#x2019;
It returned a dict with 2 keys : body and headers.
headers contains also a dict-like object which can be queried to get
the location to poll in the queue.
:param jenkins_server: The jenkins server where the job should be triggered
:return: Dict containing the response body (key body)
and the headers coming along (headers)</p>




<pre>
poll_job_in_queue(location, jenkins_server)</pre>
<p>This method poll the jenkins queue until the job is executed.
When we trigger a job through an API call,
the job is first put in the queue without having a build number assigned.
Thus we have to wait the job exit the queue to know its build number.
To do so, we have to add /api/json (or /api/xml) to the location
returned by the build_job call and poll this file.
When a &#x2018;executable&#x2019; block appears in the json, it means the job execution started
and the field &#x2018;number&#x2019; then contains the build number.
:param location: Location to poll, returned in the header of the build_job call
:param jenkins_server: The jenkins server to poll
:return: The build_number corresponding to the triggered job</p>







<pre>
class airflow.contrib.operators.jira_operator.JiraOperator(jira_conn_id=&apos;jira_default&apos;, jira_method=None, jira_method_args=None, result_processor=None, get_jira_resource_method=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>JiraOperator to interact and perform action on Jira issue tracking system.
This operator is designed to use Jira Python SDK: <a class="reference external" href="http://jira.readthedocs.io">http://jira.readthedocs.io</a></p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>jira_conn_id</strong> (<em>str</em>) &#x2013; reference to a pre-defined Jira Connection</li>
<li><strong>jira_method</strong> (<em>str</em>) &#x2013; method name from Jira Python SDK to be called</li>
<li><strong>jira_method_args</strong> (<em>dict</em>) &#x2013; required method parameters for the jira_method. (templated)</li>
<li><strong>result_processor</strong> (<em>function</em>) &#x2013; function to further process the response from Jira</li>
<li><strong>get_jira_resource_method</strong> (<em>function</em>) &#x2013; function or operator to get jira resource
on which the provided jira_method will be executed</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.contrib.operators.kubernetes_pod_operator.KubernetesPodOperator(namespace, image, name, cmds=None, arguments=None, volume_mounts=None, volumes=None, env_vars=None, secrets=None, in_cluster=False, cluster_context=None, labels=None, startup_timeout_seconds=120, get_logs=True, image_pull_policy=&apos;IfNotPresent&apos;, annotations=None, resources=None, affinity=None, config_file=None, xcom_push=False, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Execute a task in a Kubernetes Pod</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first simple">
<li><strong>image</strong> (<em>str</em>) &#x2013; Docker image you wish to launch. Defaults to dockerhub.io,
but fully qualified URLS will point to custom repositories</li>
<li><strong>cmds</strong> (<em>list of str</em>) &#x2013; entrypoint of the container. (templated)
The docker images&#x2019;s entrypoint is used if this is not provide.</li>
<li><strong>arguments</strong> (<em>list of str</em>) &#x2013; arguments of to the entrypoint. (templated)
The docker image&#x2019;s CMD is used if this is not provided.</li>
<li><strong>volume_mounts</strong> (<em>list of VolumeMount</em>) &#x2013; volumeMounts for launched pod</li>
<li><strong>volumes</strong> (<em>list of Volume</em>) &#x2013; volumes for launched pod. Includes ConfigMaps and PersistentVolumes</li>
<li><strong>labels</strong> (<em>dict</em>) &#x2013; labels to apply to the Pod</li>
<li><strong>startup_timeout_seconds</strong> (<em>int</em>) &#x2013; timeout in seconds to startup the pod</li>
<li><strong>name</strong> (<em>str</em>) &#x2013; name of the task you want to run,
will be used to generate a pod id</li>
<li><strong>env_vars</strong> (<em>dict</em>) &#x2013; Environment variables initialized in the container. (templated)</li>
<li><strong>secrets</strong> (<em>list of Secret</em>) &#x2013; Kubernetes secrets to inject in the container,
They can be exposed as environment vars or files in a volume.</li>
<li><strong>in_cluster</strong> (<em>bool</em>) &#x2013; run kubernetes client with in_cluster configuration</li>
<li><strong>cluster_context</strong> (<em>string</em>) &#x2013; context that points to kubernetes cluster.
Ignored when in_cluster is True. If None, current-context is used.</li>
<li><strong>get_logs</strong> (<em>bool</em>) &#x2013; get the stdout of the container as logs of the tasks</li>
<li><strong>affinity</strong> (<em>dict</em>) &#x2013; A dict containing a group of affinity scheduling rules</li>
<li><strong>config_file</strong> (<em>str</em>) &#x2013; The path to the Kubernetes config file</li>
<li><strong>xcom_push</strong> (<em>bool</em>) &#x2013; If xcom_push is True, the content of the file
/airflow/xcom/return.json in the container will also be pushed to an
XCom when the container completes.</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Param:</th>
<td class="field-body"><p class="first">namespace: the namespace to run within kubernetes</p>
</td>
</tr>
<tr class="field-odd field"><th class="field-name">Type:</th>
<td class="field-body"><p class="first last">namespace: str</p>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.contrib.operators.mlengine_operator.MLEngineBatchPredictionOperator(project_id, job_id, region, data_format, input_paths, output_path, model_name=None, version_name=None, uri=None, max_worker_count=None, runtime_version=None, gcp_conn_id=&apos;google_cloud_default&apos;, delegate_to=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Start a Google Cloud ML Engine prediction job.</p>
<p>NOTE: For model origin, users should consider exactly one from the
three options below:
1. Populate &#x2018;uri&#x2019; field only, which should be a GCS location that
points to a tensorflow savedModel directory.
2. Populate &#x2018;model_name&#x2019; field only, which refers to an existing
model, and the default version of the model will be used.
3. Populate both &#x2018;model_name&#x2019; and &#x2018;version_name&#x2019; fields, which
refers to a specific version of a specific model.</p>
<p>In options 2 and 3, both model and version name should contain the
minimal identifier. For instance, call</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">MLEngineBatchPredictionOperator</span><span class="p">(</span>
    <span class="o">...</span><span class="p">,</span>
    <span class="n">model_name</span><span class="o">=</span><span class="s1">&apos;my_model&apos;</span><span class="p">,</span>
    <span class="n">version_name</span><span class="o">=</span><span class="s1">&apos;my_version&apos;</span><span class="p">,</span>
    <span class="o">...</span><span class="p">)</span>
</pre>
</div>
</div>
<p>if the desired model version is
&#x201C;projects/my_project/models/my_model/versions/my_version&#x201D;.</p>
<p>See <a class="reference external" href="https://cloud.google.com/ml-engine/reference/rest/v1/projects.jobs">https://cloud.google.com/ml-engine/reference/rest/v1/projects.jobs</a>
for further documentation on the parameters.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>project_id</strong> (<em>string</em>) &#x2013; The Google Cloud project name where the
prediction job is submitted. (templated)</li>
<li><strong>job_id</strong> (<em>string</em>) &#x2013; A unique id for the prediction job on Google Cloud
ML Engine. (templated)</li>
<li><strong>data_format</strong> (<em>string</em>) &#x2013; The format of the input data.
It will default to &#x2018;DATA_FORMAT_UNSPECIFIED&#x2019; if is not provided
or is not one of [&#x201C;TEXT&#x201D;, &#x201C;TF_RECORD&#x201D;, &#x201C;TF_RECORD_GZIP&#x201D;].</li>
<li><strong>input_paths</strong> (<em>list of string</em>) &#x2013; A list of GCS paths of input data for batch
prediction. Accepting wildcard operator <a href="#id9"><span class="problematic" id="id10">*</span></a>, but only at the end. (templated)</li>
<li><strong>output_path</strong> (<em>string</em>) &#x2013; The GCS path where the prediction results are
written to. (templated)</li>
<li><strong>region</strong> (<em>string</em>) &#x2013; The Google Compute Engine region to run the
prediction job in. (templated)</li>
<li><strong>model_name</strong> (<em>string</em>) &#x2013; The Google Cloud ML Engine model to use for prediction.
If version_name is not provided, the default version of this
model will be used.
Should not be None if version_name is provided.
Should be None if uri is provided. (templated)</li>
<li><strong>version_name</strong> (<em>string</em>) &#x2013; The Google Cloud ML Engine model version to use for
prediction.
Should be None if uri is provided. (templated)</li>
<li><strong>uri</strong> (<em>string</em>) &#x2013; The GCS path of the saved model to use for prediction.
Should be None if model_name is provided.
It should be a GCS path pointing to a tensorflow SavedModel. (templated)</li>
<li><strong>max_worker_count</strong> (<em>int</em>) &#x2013; The maximum number of workers to be used
for parallel processing. Defaults to 10 if not specified.</li>
<li><strong>runtime_version</strong> (<em>string</em>) &#x2013; The Google Cloud ML Engine runtime version to use
for batch prediction.</li>
<li><strong>gcp_conn_id</strong> (<em>string</em>) &#x2013; The connection ID used for connection to Google
Cloud Platform.</li>
<li><strong>delegate_to</strong> (<em>string</em>) &#x2013; The account to impersonate, if any.
For this to work, the service account making the request must
have doamin-wide delegation enabled.</li>
</ul>
</td>
</tr>
</tbody>
</table>

<pre>Raises:</pre>
<code class="docutils literal notranslate"><span class="pre">ValueError</span></code>: if a unique model/version origin cannot be determined.





<pre>
class airflow.contrib.operators.mlengine_operator.MLEngineModelOperator(project_id, model, operation=&apos;create&apos;, gcp_conn_id=&apos;google_cloud_default&apos;, delegate_to=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Operator for managing a Google Cloud ML Engine model.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>project_id</strong> (<em>string</em>) &#x2013; The Google Cloud project name to which MLEngine
model belongs. (templated)</li>
<li><strong>model</strong> (<em>dict</em>) &#x2013; <p>A dictionary containing the information about the model.
If the <cite>operation</cite> is <cite>create</cite>, then the <cite>model</cite> parameter should
contain all the information about this model such as <cite>name</cite>.</p>
<p>If the <cite>operation</cite> is <cite>get</cite>, the <cite>model</cite> parameter
should contain the <cite>name</cite> of the model.</p>
</li>
<li><strong>operation</strong> &#x2013; <p>The operation to perform. Available operations are:</p>
<ul>
<li><code class="docutils literal notranslate"><span class="pre">create</span></code>: Creates a new model as provided by the <cite>model</cite> parameter.</li>
<li><code class="docutils literal notranslate"><span class="pre">get</span></code>: Gets a particular model where the name is specified in <cite>model</cite>.</li>
</ul>
</li>
<li><strong>gcp_conn_id</strong> (<em>string</em>) &#x2013; The connection ID to use when fetching connection info.</li>
<li><strong>delegate_to</strong> (<em>string</em>) &#x2013; The account to impersonate, if any.
For this to work, the service account making the request must have
domain-wide delegation enabled.</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.contrib.operators.mlengine_operator.MLEngineVersionOperator(project_id, model_name, version_name=None, version=None, operation=&apos;create&apos;, gcp_conn_id=&apos;google_cloud_default&apos;, delegate_to=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Operator for managing a Google Cloud ML Engine version.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>project_id</strong> (<em>string</em>) &#x2013; The Google Cloud project name to which MLEngine
model belongs.</li>
<li><strong>model_name</strong> (<em>string</em>) &#x2013; The name of the Google Cloud ML Engine model that the version
belongs to. (templated)</li>
<li><strong>version_name</strong> (<em>string</em>) &#x2013; A name to use for the version being operated upon.
If not None and the <cite>version</cite> argument is None or does not have a value for
the <cite>name</cite> key, then this will be populated in the payload for the
<cite>name</cite> key. (templated)</li>
<li><strong>version</strong> (<em>dict</em>) &#x2013; A dictionary containing the information about the version.
If the <cite>operation</cite> is <cite>create</cite>, <cite>version</cite> should contain all the
information about this version such as name, and deploymentUrl.
If the <cite>operation</cite> is <cite>get</cite> or <cite>delete</cite>, the <cite>version</cite> parameter
should contain the <cite>name</cite> of the version.
If it is None, the only <cite>operation</cite> possible would be <cite>list</cite>. (templated)</li>
<li><strong>operation</strong> (<em>string</em>) &#x2013; <p>The operation to perform. Available operations are:</p>
<ul>
<li><code class="docutils literal notranslate"><span class="pre">create</span></code>: Creates a new version in the model specified by <cite>model_name</cite>,
in which case the <cite>version</cite> parameter should contain all the
information to create that version
(e.g. <cite>name</cite>, <cite>deploymentUrl</cite>).</li>
<li><code class="docutils literal notranslate"><span class="pre">get</span></code>: Gets full information of a particular version in the model
specified by <cite>model_name</cite>.
The name of the version should be specified in the <cite>version</cite>
parameter.</li>
<li><code class="docutils literal notranslate"><span class="pre">list</span></code>: Lists all available versions of the model specified
by <cite>model_name</cite>.</li>
<li><code class="docutils literal notranslate"><span class="pre">delete</span></code>: Deletes the version specified in <cite>version</cite> parameter from the
model specified by <cite>model_name</cite>).
The name of the version should be specified in the <cite>version</cite>
parameter.</li>
</ul>
</li>
<li><strong>gcp_conn_id</strong> (<em>string</em>) &#x2013; The connection ID to use when fetching connection info.</li>
<li><strong>delegate_to</strong> (<em>string</em>) &#x2013; The account to impersonate, if any.
For this to work, the service account making the request must have
domain-wide delegation enabled.</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.contrib.operators.mlengine_operator.MLEngineTrainingOperator(project_id, job_id, package_uris, training_python_module, training_args, region, scale_tier=None, runtime_version=None, python_version=None, job_dir=None, gcp_conn_id=&apos;google_cloud_default&apos;, delegate_to=None, mode=&apos;PRODUCTION&apos;, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Operator for launching a MLEngine training job.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>project_id</strong> (<em>string</em>) &#x2013; The Google Cloud project name within which MLEngine
training job should run (templated).</li>
<li><strong>job_id</strong> (<em>string</em>) &#x2013; A unique templated id for the submitted Google MLEngine
training job. (templated)</li>
<li><strong>package_uris</strong> (<em>string</em>) &#x2013; A list of package locations for MLEngine training job,
which should include the main training program + any additional
dependencies. (templated)</li>
<li><strong>training_python_module</strong> (<em>string</em>) &#x2013; The Python module name to run within MLEngine
training job after installing &#x2018;package_uris&#x2019; packages. (templated)</li>
<li><strong>training_args</strong> (<em>string</em>) &#x2013; A list of templated command line arguments to pass to
the MLEngine training program. (templated)</li>
<li><strong>region</strong> (<em>string</em>) &#x2013; The Google Compute Engine region to run the MLEngine training
job in (templated).</li>
<li><strong>scale_tier</strong> (<em>string</em>) &#x2013; Resource tier for MLEngine training job. (templated)</li>
<li><strong>runtime_version</strong> (<em>string</em>) &#x2013; The Google Cloud ML runtime version to use for
training. (templated)</li>
<li><strong>python_version</strong> (<em>string</em>) &#x2013; The version of Python used in training. (templated)</li>
<li><strong>job_dir</strong> (<em>string</em>) &#x2013; A Google Cloud Storage path in which to store training
outputs and other data needed for training. (templated)</li>
<li><strong>gcp_conn_id</strong> (<em>string</em>) &#x2013; The connection ID to use when fetching connection info.</li>
<li><strong>delegate_to</strong> (<em>string</em>) &#x2013; The account to impersonate, if any.
For this to work, the service account making the request must have
domain-wide delegation enabled.</li>
<li><strong>mode</strong> (<em>string</em>) &#x2013; Can be one of &#x2018;DRY_RUN&#x2019;/&#x2019;CLOUD&#x2019;. In &#x2018;DRY_RUN&#x2019; mode, no real
training job will be launched, but the MLEngine training job request
will be printed out. In &#x2018;CLOUD&#x2019; mode, a real MLEngine training job
creation request will be issued.</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.contrib.operators.mongo_to_s3.MongoToS3Operator(mongo_conn_id, s3_conn_id, mongo_collection, mongo_query, s3_bucket, s3_key, mongo_db=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>

<pre>Mongo -&gt; S3</pre>
<p class="first">A more specific baseOperator meant to move data
from mongo via pymongo to s3 via boto</p>

<pre>things to note</pre>
.execute() is written to depend on .transform()
.transform() is meant to be extended by child classes
to perform transformations unique to those operators needs




<pre>
execute(context)</pre>
<p>Executed by task_instance at runtime</p>




<pre>
transform(docs)</pre>

<pre>Processes pyMongo cursor and returns an iterable with each element being</pre>
a JSON serializable dictionary

<p>Base transform() assumes no processing is needed
ie. docs is a pyMongo cursor of documents and cursor just
needs to be passed through</p>
<p>Override this method for custom transformations</p>







<pre>
class airflow.contrib.operators.mysql_to_gcs.MySqlToGoogleCloudStorageOperator(sql, bucket, filename, schema_filename=None, approx_max_file_size_bytes=1900000000, mysql_conn_id=&apos;mysql_default&apos;, google_cloud_storage_conn_id=&apos;google_cloud_default&apos;, schema=None, delegate_to=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Copy data from MySQL to Google cloud storage in JSON format.</p>

<pre>
classmethod type_map(mysql_type)</pre>
<p>Helper function that maps from MySQL fields to BigQuery fields. Used
when a schema_filename is set.</p>







<pre>
class airflow.contrib.operators.postgres_to_gcs_operator.PostgresToGoogleCloudStorageOperator(sql, bucket, filename, schema_filename=None, approx_max_file_size_bytes=1900000000, postgres_conn_id=&apos;postgres_default&apos;, google_cloud_storage_conn_id=&apos;google_cloud_default&apos;, delegate_to=None, parameters=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Copy data from Postgres to Google Cloud Storage in JSON format.</p>

<pre>
classmethod convert_types(value)</pre>
<p>Takes a value from Postgres, and converts it to a value that&#x2019;s safe for
JSON/Google Cloud Storage/BigQuery. Dates are converted to UTC seconds.
Decimals are converted to floats. Times are converted to seconds.</p>




<pre>
classmethod type_map(postgres_type)</pre>
<p>Helper function that maps from Postgres fields to BigQuery fields. Used
when a schema_filename is set.</p>







<pre>
class airflow.contrib.operators.pubsub_operator.PubSubTopicCreateOperator(project, topic, fail_if_exists=False, gcp_conn_id=&apos;google_cloud_default&apos;, delegate_to=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Create a PubSub topic.</p>
<p>By default, if the topic already exists, this operator will
not cause the DAG to fail.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="k">with</span> <span class="n">DAG</span><span class="p">(</span><span class="s1">&apos;successful DAG&apos;</span><span class="p">)</span> <span class="k">as</span> <span class="n">dag</span><span class="p">:</span>
    <span class="p">(</span>
        <span class="n">dag</span>
        <span class="o">&gt;&gt;</span> <span class="n">PubSubTopicCreateOperator</span><span class="p">(</span><span class="n">project</span><span class="o">=</span><span class="s1">&apos;my-project&apos;</span><span class="p">,</span>
                                     <span class="n">topic</span><span class="o">=</span><span class="s1">&apos;my_new_topic&apos;</span><span class="p">)</span>
        <span class="o">&gt;&gt;</span> <span class="n">PubSubTopicCreateOperator</span><span class="p">(</span><span class="n">project</span><span class="o">=</span><span class="s1">&apos;my-project&apos;</span><span class="p">,</span>
                                     <span class="n">topic</span><span class="o">=</span><span class="s1">&apos;my_new_topic&apos;</span><span class="p">)</span>
    <span class="p">)</span>
</pre>
</div>
</div>
<p>The operator can be configured to fail if the topic already exists.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="k">with</span> <span class="n">DAG</span><span class="p">(</span><span class="s1">&apos;failing DAG&apos;</span><span class="p">)</span> <span class="k">as</span> <span class="n">dag</span><span class="p">:</span>
    <span class="p">(</span>
        <span class="n">dag</span>
        <span class="o">&gt;&gt;</span> <span class="n">PubSubTopicCreateOperator</span><span class="p">(</span><span class="n">project</span><span class="o">=</span><span class="s1">&apos;my-project&apos;</span><span class="p">,</span>
                                     <span class="n">topic</span><span class="o">=</span><span class="s1">&apos;my_new_topic&apos;</span><span class="p">)</span>
        <span class="o">&gt;&gt;</span> <span class="n">PubSubTopicCreateOperator</span><span class="p">(</span><span class="n">project</span><span class="o">=</span><span class="s1">&apos;my-project&apos;</span><span class="p">,</span>
                                     <span class="n">topic</span><span class="o">=</span><span class="s1">&apos;my_new_topic&apos;</span><span class="p">,</span>
                                     <span class="n">fail_if_exists</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
    <span class="p">)</span>
</pre>
</div>
</div>
<p>Both <code class="docutils literal notranslate"><span class="pre">project</span></code> and <code class="docutils literal notranslate"><span class="pre">topic</span></code> are templated so you can use
variables in them.</p>




<pre>
class airflow.contrib.operators.pubsub_operator.PubSubTopicDeleteOperator(project, topic, fail_if_not_exists=False, gcp_conn_id=&apos;google_cloud_default&apos;, delegate_to=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Delete a PubSub topic.</p>
<p>By default, if the topic does not exist, this operator will
not cause the DAG to fail.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="k">with</span> <span class="n">DAG</span><span class="p">(</span><span class="s1">&apos;successful DAG&apos;</span><span class="p">)</span> <span class="k">as</span> <span class="n">dag</span><span class="p">:</span>
    <span class="p">(</span>
        <span class="n">dag</span>
        <span class="o">&gt;&gt;</span> <span class="n">PubSubTopicDeleteOperator</span><span class="p">(</span><span class="n">project</span><span class="o">=</span><span class="s1">&apos;my-project&apos;</span><span class="p">,</span>
                                     <span class="n">topic</span><span class="o">=</span><span class="s1">&apos;non_existing_topic&apos;</span><span class="p">)</span>
    <span class="p">)</span>
</pre>
</div>
</div>
<p>The operator can be configured to fail if the topic does not exist.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="k">with</span> <span class="n">DAG</span><span class="p">(</span><span class="s1">&apos;failing DAG&apos;</span><span class="p">)</span> <span class="k">as</span> <span class="n">dag</span><span class="p">:</span>
    <span class="p">(</span>
        <span class="n">dag</span>
        <span class="o">&gt;&gt;</span> <span class="n">PubSubTopicCreateOperator</span><span class="p">(</span><span class="n">project</span><span class="o">=</span><span class="s1">&apos;my-project&apos;</span><span class="p">,</span>
                                     <span class="n">topic</span><span class="o">=</span><span class="s1">&apos;non_existing_topic&apos;</span><span class="p">,</span>
                                     <span class="n">fail_if_not_exists</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
    <span class="p">)</span>
</pre>
</div>
</div>
<p>Both <code class="docutils literal notranslate"><span class="pre">project</span></code> and <code class="docutils literal notranslate"><span class="pre">topic</span></code> are templated so you can use
variables in them.</p>




<pre>
class airflow.contrib.operators.pubsub_operator.PubSubSubscriptionCreateOperator(topic_project, topic, subscription=None, subscription_project=None, ack_deadline_secs=10, fail_if_exists=False, gcp_conn_id=&apos;google_cloud_default&apos;, delegate_to=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Create a PubSub subscription.</p>
<p>By default, the subscription will be created in <code class="docutils literal notranslate"><span class="pre">topic_project</span></code>. If
<code class="docutils literal notranslate"><span class="pre">subscription_project</span></code> is specified and the GCP credentials allow, the
Subscription can be created in a different project from its topic.</p>
<p>By default, if the subscription already exists, this operator will
not cause the DAG to fail. However, the topic must exist in the project.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="k">with</span> <span class="n">DAG</span><span class="p">(</span><span class="s1">&apos;successful DAG&apos;</span><span class="p">)</span> <span class="k">as</span> <span class="n">dag</span><span class="p">:</span>
    <span class="p">(</span>
        <span class="n">dag</span>
        <span class="o">&gt;&gt;</span> <span class="n">PubSubSubscriptionCreateOperator</span><span class="p">(</span>
            <span class="n">topic_project</span><span class="o">=</span><span class="s1">&apos;my-project&apos;</span><span class="p">,</span> <span class="n">topic</span><span class="o">=</span><span class="s1">&apos;my-topic&apos;</span><span class="p">,</span>
            <span class="n">subscription</span><span class="o">=</span><span class="s1">&apos;my-subscription&apos;</span><span class="p">)</span>
        <span class="o">&gt;&gt;</span> <span class="n">PubSubSubscriptionCreateOperator</span><span class="p">(</span>
            <span class="n">topic_project</span><span class="o">=</span><span class="s1">&apos;my-project&apos;</span><span class="p">,</span> <span class="n">topic</span><span class="o">=</span><span class="s1">&apos;my-topic&apos;</span><span class="p">,</span>
            <span class="n">subscription</span><span class="o">=</span><span class="s1">&apos;my-subscription&apos;</span><span class="p">)</span>
    <span class="p">)</span>
</pre>
</div>
</div>
<p>The operator can be configured to fail if the subscription already exists.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="k">with</span> <span class="n">DAG</span><span class="p">(</span><span class="s1">&apos;failing DAG&apos;</span><span class="p">)</span> <span class="k">as</span> <span class="n">dag</span><span class="p">:</span>
    <span class="p">(</span>
        <span class="n">dag</span>
        <span class="o">&gt;&gt;</span> <span class="n">PubSubSubscriptionCreateOperator</span><span class="p">(</span>
            <span class="n">topic_project</span><span class="o">=</span><span class="s1">&apos;my-project&apos;</span><span class="p">,</span> <span class="n">topic</span><span class="o">=</span><span class="s1">&apos;my-topic&apos;</span><span class="p">,</span>
            <span class="n">subscription</span><span class="o">=</span><span class="s1">&apos;my-subscription&apos;</span><span class="p">)</span>
        <span class="o">&gt;&gt;</span> <span class="n">PubSubSubscriptionCreateOperator</span><span class="p">(</span>
            <span class="n">topic_project</span><span class="o">=</span><span class="s1">&apos;my-project&apos;</span><span class="p">,</span> <span class="n">topic</span><span class="o">=</span><span class="s1">&apos;my-topic&apos;</span><span class="p">,</span>
            <span class="n">subscription</span><span class="o">=</span><span class="s1">&apos;my-subscription&apos;</span><span class="p">,</span> <span class="n">fail_if_exists</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
    <span class="p">)</span>
</pre>
</div>
</div>
<p>Finally, subscription is not required. If not passed, the operator will
generated a universally unique identifier for the subscription&#x2019;s name.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="k">with</span> <span class="n">DAG</span><span class="p">(</span><span class="s1">&apos;DAG&apos;</span><span class="p">)</span> <span class="k">as</span> <span class="n">dag</span><span class="p">:</span>
    <span class="p">(</span>
        <span class="n">dag</span> <span class="o">&gt;&gt;</span> <span class="n">PubSubSubscriptionCreateOperator</span><span class="p">(</span>
            <span class="n">topic_project</span><span class="o">=</span><span class="s1">&apos;my-project&apos;</span><span class="p">,</span> <span class="n">topic</span><span class="o">=</span><span class="s1">&apos;my-topic&apos;</span><span class="p">)</span>
    <span class="p">)</span>
</pre>
</div>
</div>
<p><code class="docutils literal notranslate"><span class="pre">topic_project</span></code>, <code class="docutils literal notranslate"><span class="pre">topic</span></code>, <code class="docutils literal notranslate"><span class="pre">subscription</span></code>, and
<code class="docutils literal notranslate"><span class="pre">subscription</span></code> are templated so you can use variables in them.</p>




<pre>
class airflow.contrib.operators.pubsub_operator.PubSubSubscriptionDeleteOperator(project, subscription, fail_if_not_exists=False, gcp_conn_id=&apos;google_cloud_default&apos;, delegate_to=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Delete a PubSub subscription.</p>
<p>By default, if the subscription does not exist, this operator will
not cause the DAG to fail.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="k">with</span> <span class="n">DAG</span><span class="p">(</span><span class="s1">&apos;successful DAG&apos;</span><span class="p">)</span> <span class="k">as</span> <span class="n">dag</span><span class="p">:</span>
    <span class="p">(</span>
        <span class="n">dag</span>
        <span class="o">&gt;&gt;</span> <span class="n">PubSubSubscriptionDeleteOperator</span><span class="p">(</span><span class="n">project</span><span class="o">=</span><span class="s1">&apos;my-project&apos;</span><span class="p">,</span>
                                            <span class="n">subscription</span><span class="o">=</span><span class="s1">&apos;non-existing&apos;</span><span class="p">)</span>
    <span class="p">)</span>
</pre>
</div>
</div>
<p>The operator can be configured to fail if the subscription already exists.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="k">with</span> <span class="n">DAG</span><span class="p">(</span><span class="s1">&apos;failing DAG&apos;</span><span class="p">)</span> <span class="k">as</span> <span class="n">dag</span><span class="p">:</span>
    <span class="p">(</span>
        <span class="n">dag</span>
        <span class="o">&gt;&gt;</span> <span class="n">PubSubSubscriptionDeleteOperator</span><span class="p">(</span>
             <span class="n">project</span><span class="o">=</span><span class="s1">&apos;my-project&apos;</span><span class="p">,</span> <span class="n">subscription</span><span class="o">=</span><span class="s1">&apos;non-existing&apos;</span><span class="p">,</span>
             <span class="n">fail_if_not_exists</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
    <span class="p">)</span>
</pre>
</div>
</div>
<p><code class="docutils literal notranslate"><span class="pre">project</span></code>, and <code class="docutils literal notranslate"><span class="pre">subscription</span></code> are templated so you can use
variables in them.</p>




<pre>
class airflow.contrib.operators.pubsub_operator.PubSubPublishOperator(project, topic, messages, gcp_conn_id=&apos;google_cloud_default&apos;, delegate_to=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Publish messages to a PubSub topic.</p>
<p>Each Task publishes all provided messages to the same topic
in a single GCP project. If the topic does not exist, this
task will fail.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>   from base64 import b64encode as b64e

   m1 = {&apos;data&apos;: b64e(&apos;Hello, World!&apos;),
         &apos;attributes&apos;: {&apos;type&apos;: &apos;greeting&apos;}
        }
   m2 = {&apos;data&apos;: b64e(&apos;Knock, knock&apos;)}
   m3 = {&apos;attributes&apos;: {&apos;foo&apos;: &apos;&apos;}}

   t1 = PubSubPublishOperator(
       project=&apos;my-project&apos;,topic=&apos;my_topic&apos;,
       messages=[m1, m2, m3],
       create_topic=True,
       dag=dag)

``project`` , ``topic``, and ``messages`` are templated so you can use
</pre>
</div>
</div>
<p>variables in them.</p>




<pre>
class airflow.contrib.operators.qubole_check_operator.QuboleCheckOperator(qubole_conn_id=&apos;qubole_default&apos;, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.operators.check_operator.CheckOperator" title="airflow.operators.check_operator.CheckOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.operators.check_operator.CheckOperator</span></code></a>, <a class="reference internal" href="#airflow.contrib.operators.qubole_operator.QuboleOperator" title="airflow.contrib.operators.qubole_operator.QuboleOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.contrib.operators.qubole_operator.QuboleOperator</span></code></a></p>
<p>Performs checks against Qubole Commands. <code class="docutils literal notranslate"><span class="pre">QuboleCheckOperator</span></code> expects
a command that will be executed on QDS.
By default, each value on first row of the result of this Qubole Commmand
is evaluated using python <code class="docutils literal notranslate"><span class="pre">bool</span></code> casting. If any of the
values return <code class="docutils literal notranslate"><span class="pre">False</span></code>, the check is failed and errors out.</p>
<p>Note that Python bool casting evals the following as <code class="docutils literal notranslate"><span class="pre">False</span></code>:</p>
<ul class="simple">
<li><code class="docutils literal notranslate"><span class="pre">False</span></code></li>
<li><code class="docutils literal notranslate"><span class="pre">0</span></code></li>
<li>Empty string (<code class="docutils literal notranslate"><span class="pre">&quot;&quot;</span></code>)</li>
<li>Empty list (<code class="docutils literal notranslate"><span class="pre">[]</span></code>)</li>
<li>Empty dictionary or set (<code class="docutils literal notranslate"><span class="pre">{}</span></code>)</li>
</ul>
<p>Given a query like <code class="docutils literal notranslate"><span class="pre">SELECT</span> <span class="pre">COUNT(*)</span> <span class="pre">FROM</span> <span class="pre">foo</span></code>, it will fail only if
the count <code class="docutils literal notranslate"><span class="pre">==</span> <span class="pre">0</span></code>. You can craft much more complex query that could,
for instance, check that the table has the same number of rows as
the source table upstream, or that the count of today&#x2019;s partition is
greater than yesterday&#x2019;s partition, or that a set of metrics are less
than 3 standard deviation for the 7 day average.</p>
<p>This operator can be used as a data quality check in your pipeline, and
depending on where you put it in your DAG, you have the choice to
stop the critical path, preventing from
publishing dubious data, or on the side and receive email alerts
without stopping the progress of the DAG.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><strong>qubole_conn_id</strong> (<em>str</em>) &#x2013; Connection id which consists of qds auth_token</td>
</tr>
</tbody>
</table>
<p>kwargs:</p>
<blockquote>
<div><p>Arguments specific to Qubole command can be referred from QuboleOperator docs.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name" colspan="2">results_parser_callable:</th>
</tr>
<tr class="field-odd field"><td>&#xA0;</td>
<td class="field-body">This is an optional parameter to
extend the flexibility of parsing the results of Qubole
command to the users. This is a python callable which
can hold the logic to parse list of rows returned by Qubole command.
By default, only the values on first row are used for performing checks.
This callable should return a list of records on
which the checks have to be performed.</td>
</tr>
</tbody>
</table>
</div>
</blockquote>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">All fields in common with template fields of
QuboleOperator and CheckOperator are template-supported.</p>
</div>




<pre>
class airflow.contrib.operators.qubole_check_operator.QuboleValueCheckOperator(pass_value, tolerance=None, qubole_conn_id=&apos;qubole_default&apos;, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.operators.check_operator.ValueCheckOperator" title="airflow.operators.check_operator.ValueCheckOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.operators.check_operator.ValueCheckOperator</span></code></a>, <a class="reference internal" href="#airflow.contrib.operators.qubole_operator.QuboleOperator" title="airflow.contrib.operators.qubole_operator.QuboleOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.contrib.operators.qubole_operator.QuboleOperator</span></code></a></p>
<p>Performs a simple value check using Qubole command.
By default, each value on the first row of this
Qubole command is compared with a pre-defined value.
The check fails and errors out if the output of the command
is not within the permissible limit of expected value.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>qubole_conn_id</strong> (<em>str</em>) &#x2013; Connection id which consists of qds auth_token</li>
<li><strong>pass_value</strong> (<em>str/int/float</em>) &#x2013; Expected value of the query results.</li>
<li><strong>tolerance</strong> (<em>int/float</em>) &#x2013; Defines the permissible pass_value range, for example if
tolerance is 2, the Qubole command output can be anything between
-2*pass_value and 2*pass_value, without the operator erring out.</li>
</ul>
</td>
</tr>
</tbody>
</table>
<p>kwargs:</p>
<blockquote>
<div><p>Arguments specific to Qubole command can be referred from QuboleOperator docs.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name" colspan="2">results_parser_callable:</th>
</tr>
<tr class="field-odd field"><td>&#xA0;</td>
<td class="field-body">This is an optional parameter to
extend the flexibility of parsing the results of Qubole
command to the users. This is a python callable which
can hold the logic to parse list of rows returned by Qubole command.
By default, only the values on first row are used for performing checks.
This callable should return a list of records on
which the checks have to be performed.</td>
</tr>
</tbody>
</table>
</div>
</blockquote>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">All fields in common with template fields of
QuboleOperator and ValueCheckOperator are template-supported.</p>
</div>




<pre>
class airflow.contrib.operators.qubole_operator.QuboleOperator(qubole_conn_id=&apos;qubole_default&apos;, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Execute tasks (commands) on QDS (<a class="reference external" href="https://qubole.com">https://qubole.com</a>).</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><strong>qubole_conn_id</strong> (<em>str</em>) &#x2013; Connection id which consists of qds auth_token</td>
</tr>
</tbody>
</table>

<pre>kwargs:</pre>
<table class="first docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">command_type:</th>
<td class="field-body">type of command to be executed, e.g. hivecmd, shellcmd, hadoopcmd</td>
</tr>
<tr class="field-even field"><th class="field-name">tags:</th>
<td class="field-body">array of tags to be assigned with the command</td>
</tr>
<tr class="field-odd field"><th class="field-name">cluster_label:</th>
<td class="field-body">cluster label on which the command will be executed</td>
</tr>
<tr class="field-even field"><th class="field-name">name:</th>
<td class="field-body">name to be given to command</td>
</tr>
<tr class="field-odd field"><th class="field-name">notify:</th>
<td class="field-body">whether to send email on command completion or not (default is False)</td>
</tr>
</tbody>
</table>
<p><strong>Arguments specific to command types</strong></p>

<pre>hivecmd:</pre>
<table class="first last docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">query:</th>
<td class="field-body">inline query statement</td>
</tr>
<tr class="field-even field"><th class="field-name" colspan="2">script_location:</th>
</tr>
<tr class="field-even field"><td>&#xA0;</td>
<td class="field-body">s3 location containing query statement</td>
</tr>
<tr class="field-odd field"><th class="field-name">sample_size:</th>
<td class="field-body">size of sample in bytes on which to run query</td>
</tr>
<tr class="field-even field"><th class="field-name">macros:</th>
<td class="field-body">macro values which were used in query</td>
</tr>
</tbody>
</table>

<pre>prestocmd:</pre>
<table class="first last docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">query:</th>
<td class="field-body">inline query statement</td>
</tr>
<tr class="field-even field"><th class="field-name" colspan="2">script_location:</th>
</tr>
<tr class="field-even field"><td>&#xA0;</td>
<td class="field-body">s3 location containing query statement</td>
</tr>
<tr class="field-odd field"><th class="field-name">macros:</th>
<td class="field-body">macro values which were used in query</td>
</tr>
</tbody>
</table>

<pre>hadoopcmd:</pre>
<table class="first last docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">sub_commnad:</th>
<td class="field-body">must be one these [&#x201C;jar&#x201D;, &#x201C;s3distcp&#x201D;, &#x201C;streaming&#x201D;] followed by
1 or more args</td>
</tr>
</tbody>
</table>

<pre>shellcmd:</pre>
<table class="first last docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">script:</th>
<td class="field-body">inline command with args</td>
</tr>
<tr class="field-even field"><th class="field-name" colspan="2">script_location:</th>
</tr>
<tr class="field-even field"><td>&#xA0;</td>
<td class="field-body">s3 location containing query statement</td>
</tr>
<tr class="field-odd field"><th class="field-name">files:</th>
<td class="field-body">list of files in s3 bucket as file1,file2 format. These files will be
copied into the working directory where the qubole command is being
executed.</td>
</tr>
<tr class="field-even field"><th class="field-name">archives:</th>
<td class="field-body">list of archives in s3 bucket as archive1,archive2 format. These
will be unarchived intothe working directory where the qubole command is
being executed</td>
</tr>
<tr class="field-odd field"><th class="field-name">parameters:</th>
<td class="field-body">any extra args which need to be passed to script (only when
script_location is supplied)</td>
</tr>
</tbody>
</table>

<pre>pigcmd:</pre>
<table class="first last docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">script:</th>
<td class="field-body">inline query statement (latin_statements)</td>
</tr>
<tr class="field-even field"><th class="field-name" colspan="2">script_location:</th>
</tr>
<tr class="field-even field"><td>&#xA0;</td>
<td class="field-body">s3 location containing pig query</td>
</tr>
<tr class="field-odd field"><th class="field-name">parameters:</th>
<td class="field-body">any extra args which need to be passed to script (only when
script_location is supplied</td>
</tr>
</tbody>
</table>

<pre>sparkcmd:</pre>
<table class="first last docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">program:</th>
<td class="field-body">the complete Spark Program in Scala, SQL, Command, R, or Python</td>
</tr>
<tr class="field-even field"><th class="field-name">cmdline:</th>
<td class="field-body">spark-submit command line, all required information must be specify
in cmdline itself.</td>
</tr>
<tr class="field-odd field"><th class="field-name">sql:</th>
<td class="field-body">inline sql query</td>
</tr>
<tr class="field-even field"><th class="field-name" colspan="2">script_location:</th>
</tr>
<tr class="field-even field"><td>&#xA0;</td>
<td class="field-body">s3 location containing query statement</td>
</tr>
<tr class="field-odd field"><th class="field-name">language:</th>
<td class="field-body">language of the program, Scala, SQL, Command, R, or Python</td>
</tr>
<tr class="field-even field"><th class="field-name">app_id:</th>
<td class="field-body">ID of an Spark job server app</td>
</tr>
<tr class="field-odd field"><th class="field-name">arguments:</th>
<td class="field-body">spark-submit command line arguments</td>
</tr>
<tr class="field-even field"><th class="field-name" colspan="2">user_program_arguments:</th>
</tr>
<tr class="field-even field"><td>&#xA0;</td>
<td class="field-body">arguments that the user program takes in</td>
</tr>
<tr class="field-odd field"><th class="field-name">macros:</th>
<td class="field-body">macro values which were used in query</td>
</tr>
</tbody>
</table>

<pre>dbtapquerycmd:</pre>
<table class="first last docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">db_tap_id:</th>
<td class="field-body">data store ID of the target database, in Qubole.</td>
</tr>
<tr class="field-even field"><th class="field-name">query:</th>
<td class="field-body">inline query statement</td>
</tr>
<tr class="field-odd field"><th class="field-name">macros:</th>
<td class="field-body">macro values which were used in query</td>
</tr>
</tbody>
</table>

<pre>dbexportcmd:</pre>
<table class="first last docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">mode:</th>
<td class="field-body">1 (simple), 2 (advance)</td>
</tr>
<tr class="field-even field"><th class="field-name">hive_table:</th>
<td class="field-body">Name of the hive table</td>
</tr>
<tr class="field-odd field"><th class="field-name">partition_spec:</th>
<td class="field-body">partition specification for Hive table.</td>
</tr>
<tr class="field-even field"><th class="field-name">dbtap_id:</th>
<td class="field-body">data store ID of the target database, in Qubole.</td>
</tr>
<tr class="field-odd field"><th class="field-name">db_table:</th>
<td class="field-body">name of the db table</td>
</tr>
<tr class="field-even field"><th class="field-name">db_update_mode:</th>
<td class="field-body">allowinsert or updateonly</td>
</tr>
<tr class="field-odd field"><th class="field-name">db_update_keys:</th>
<td class="field-body">columns used to determine the uniqueness of rows</td>
</tr>
<tr class="field-even field"><th class="field-name">export_dir:</th>
<td class="field-body">HDFS/S3 location from which data will be exported.</td>
</tr>
<tr class="field-odd field"><th class="field-name" colspan="2">fields_terminated_by:</th>
</tr>
<tr class="field-odd field"><td>&#xA0;</td>
<td class="field-body">hex of the char used as column separator in the dataset</td>
</tr>
</tbody>
</table>

<pre>dbimportcmd:</pre>
<table class="first last docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">mode:</th>
<td class="field-body">1 (simple), 2 (advance)</td>
</tr>
<tr class="field-even field"><th class="field-name">hive_table:</th>
<td class="field-body">Name of the hive table</td>
</tr>
<tr class="field-odd field"><th class="field-name">dbtap_id:</th>
<td class="field-body">data store ID of the target database, in Qubole.</td>
</tr>
<tr class="field-even field"><th class="field-name">db_table:</th>
<td class="field-body">name of the db table</td>
</tr>
<tr class="field-odd field"><th class="field-name">where_clause:</th>
<td class="field-body">where clause, if any</td>
</tr>
<tr class="field-even field"><th class="field-name">parallelism:</th>
<td class="field-body">number of parallel db connections to use for extracting data</td>
</tr>
<tr class="field-odd field"><th class="field-name">extract_query:</th>
<td class="field-body">SQL query to extract data from db. $CONDITIONS must be part
of the where clause.</td>
</tr>
<tr class="field-even field"><th class="field-name">boundary_query:</th>
<td class="field-body">Query to be used get range of row IDs to be extracted</td>
</tr>
<tr class="field-odd field"><th class="field-name">split_column:</th>
<td class="field-body">Column used as row ID to split data into ranges (mode 2)</td>
</tr>
</tbody>
</table>




<div class="admonition note">
<p class="first admonition-title">Note</p>
<p>Following fields are template-supported : <code class="docutils literal notranslate"><span class="pre">query</span></code>, <code class="docutils literal notranslate"><span class="pre">script_location</span></code>,
<code class="docutils literal notranslate"><span class="pre">sub_command</span></code>, <code class="docutils literal notranslate"><span class="pre">script</span></code>, <code class="docutils literal notranslate"><span class="pre">files</span></code>, <code class="docutils literal notranslate"><span class="pre">archives</span></code>, <code class="docutils literal notranslate"><span class="pre">program</span></code>, <code class="docutils literal notranslate"><span class="pre">cmdline</span></code>,
<code class="docutils literal notranslate"><span class="pre">sql</span></code>, <code class="docutils literal notranslate"><span class="pre">where_clause</span></code>, <code class="docutils literal notranslate"><span class="pre">extract_query</span></code>, <code class="docutils literal notranslate"><span class="pre">boundary_query</span></code>, <code class="docutils literal notranslate"><span class="pre">macros</span></code>,
<code class="docutils literal notranslate"><span class="pre">tags</span></code>, <code class="docutils literal notranslate"><span class="pre">name</span></code>, <code class="docutils literal notranslate"><span class="pre">parameters</span></code>, <code class="docutils literal notranslate"><span class="pre">dbtap_id</span></code>, <code class="docutils literal notranslate"><span class="pre">hive_table</span></code>, <code class="docutils literal notranslate"><span class="pre">db_table</span></code>,
<code class="docutils literal notranslate"><span class="pre">split_column</span></code>, <code class="docutils literal notranslate"><span class="pre">note_id</span></code>, <code class="docutils literal notranslate"><span class="pre">db_update_keys</span></code>, <code class="docutils literal notranslate"><span class="pre">export_dir</span></code>,
<code class="docutils literal notranslate"><span class="pre">partition_spec</span></code>, <code class="docutils literal notranslate"><span class="pre">qubole_conn_id</span></code>, <code class="docutils literal notranslate"><span class="pre">arguments</span></code>, <code class="docutils literal notranslate"><span class="pre">user_program_arguments</span></code>.</p>
<blockquote class="last">
<div>You can also use <code class="docutils literal notranslate"><span class="pre">.txt</span></code> files for template driven use cases.</div>
</blockquote>
</div>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">In QuboleOperator there is a default handler for task failures and retries,
which generally kills the command running at QDS for the corresponding task
instance. You can override this behavior by providing your own failure and retry
handler in task definition.</p>
</div>




<pre>
class airflow.contrib.operators.s3_list_operator.S3ListOperator(bucket, prefix=&apos;&apos;, delimiter=&apos;&apos;, aws_conn_id=&apos;aws_default&apos;, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>List all objects from the bucket with the given string prefix in name.</p>
<p>This operator returns a python list with the name of objects which can be
used by <cite>xcom</cite> in the downstream task.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>bucket</strong> (<em>string</em>) &#x2013; The S3 bucket where to find the objects. (templated)</li>
<li><strong>prefix</strong> (<em>string</em>) &#x2013; Prefix string to filters the objects whose name begin with
such prefix. (templated)</li>
<li><strong>delimiter</strong> (<em>string</em>) &#x2013; the delimiter marks key hierarchy. (templated)</li>
<li><strong>aws_conn_id</strong> (<em>string</em>) &#x2013; The connection ID to use when connecting to S3 storage.</li>
</ul>
</td>
</tr>
</tbody>
</table>

<pre>Example:</pre>
<p class="first">The following operator would list all the files
(excluding subfolders) from the S3
<code class="docutils literal notranslate"><span class="pre">customers/2018/04/</span></code> key in the <code class="docutils literal notranslate"><span class="pre">data</span></code> bucket.</p>
<div class="last highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">s3_file</span> <span class="o">=</span> <span class="n">S3ListOperator</span><span class="p">(</span>
    <span class="n">task_id</span><span class="o">=</span><span class="s1">&apos;list_3s_files&apos;</span><span class="p">,</span>
    <span class="n">bucket</span><span class="o">=</span><span class="s1">&apos;data&apos;</span><span class="p">,</span>
    <span class="n">prefix</span><span class="o">=</span><span class="s1">&apos;customers/2018/04/&apos;</span><span class="p">,</span>
    <span class="n">delimiter</span><span class="o">=</span><span class="s1">&apos;/&apos;</span><span class="p">,</span>
    <span class="n">aws_conn_id</span><span class="o">=</span><span class="s1">&apos;aws_customers_conn&apos;</span>
<span class="p">)</span>
</pre>
</div>
</div>






<pre>
class airflow.contrib.operators.s3_to_gcs_operator.S3ToGoogleCloudStorageOperator(bucket, prefix=&apos;&apos;, delimiter=&apos;&apos;, aws_conn_id=&apos;aws_default&apos;, dest_gcs_conn_id=None, dest_gcs=None, delegate_to=None, replace=False, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="integration.html#airflow.contrib.operators.s3_list_operator.S3ListOperator" title="airflow.contrib.operators.s3_list_operator.S3ListOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.contrib.operators.s3_list_operator.S3ListOperator</span></code></a></p>
<p>Synchronizes an S3 key, possibly a prefix, with a Google Cloud Storage
destination path.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>bucket</strong> (<em>string</em>) &#x2013; The S3 bucket where to find the objects. (templated)</li>
<li><strong>prefix</strong> (<em>string</em>) &#x2013; Prefix string which filters objects whose name begin with
such prefix. (templated)</li>
<li><strong>delimiter</strong> (<em>string</em>) &#x2013; the delimiter marks key hierarchy. (templated)</li>
<li><strong>aws_conn_id</strong> (<em>string</em>) &#x2013; The source S3 connection</li>
<li><strong>dest_gcs_conn_id</strong> (<em>string</em>) &#x2013; The destination connection ID to use
when connecting to Google Cloud Storage.</li>
<li><strong>dest_gcs</strong> (<em>string</em>) &#x2013; The destination Google Cloud Storage bucket and prefix
where you want to store the files. (templated)</li>
<li><strong>delegate_to</strong> (<em>string</em>) &#x2013; The account to impersonate, if any.
For this to work, the service account making the request must have
domain-wide delegation enabled.</li>
<li><strong>replace</strong> (<em>bool</em>) &#x2013; Whether you want to replace existing destination files
or not.</li>
</ul>
</td>
</tr>
</tbody>
</table>
<p><strong>Example</strong>:
.. code-block:: python</p>
<blockquote>
<div>
<pre>s3_to_gcs_op = S3ToGoogleCloudStorageOperator(</pre>
task_id=&#x2019;s3_to_gcs_example&#x2019;,
bucket=&#x2019;my-s3-bucket&#x2019;,
prefix=&#x2019;data/customers-201804&#x2019;,
dest_gcs_conn_id=&#x2019;google_cloud_default&#x2019;,
dest_gcs=&#x2019;gs://my.gcs.bucket/some/customers/&#x2019;,
replace=False,
dag=my-dag)

</div>
</blockquote>
<p>Note that <code class="docutils literal notranslate"><span class="pre">bucket</span></code>, <code class="docutils literal notranslate"><span class="pre">prefix</span></code>, <code class="docutils literal notranslate"><span class="pre">delimiter</span></code> and <code class="docutils literal notranslate"><span class="pre">dest_gcs</span></code> are
templated, so you can use variables in them if you wish.</p>




<pre>
class airflow.contrib.operators.segment_track_event_operator.SegmentTrackEventOperator(user_id, event, properties=None, segment_conn_id=&apos;segment_default&apos;, segment_debug_mode=False, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Send Track Event to Segment for a specified user_id and event</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>user_id</strong> (<em>string</em>) &#x2013; The ID for this user in your database. (templated)</li>
<li><strong>event</strong> (<em>string</em>) &#x2013; The name of the event you&#x2019;re tracking. (templated)</li>
<li><strong>properties</strong> (<em>dict</em>) &#x2013; A dictionary of properties for the event. (templated)</li>
<li><strong>segment_conn_id</strong> (<em>string</em>) &#x2013; The connection ID to use when connecting to Segment.</li>
<li><strong>segment_debug_mode</strong> (<em>boolean</em>) &#x2013; Determines whether Segment should run in debug mode.
Defaults to False</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.contrib.operators.sftp_operator.SFTPOperator(ssh_hook=None, ssh_conn_id=None, remote_host=None, local_filepath=None, remote_filepath=None, operation=&apos;put&apos;, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>SFTPOperator for transferring files from remote host to local or vice a versa.
This operator uses ssh_hook to open sftp trasport channel that serve as basis
for file transfer.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>ssh_hook</strong> (<code class="xref py py-class docutils literal notranslate"><span class="pre">SSHHook</span></code>) &#x2013; predefined ssh_hook to use for remote execution</li>
<li><strong>ssh_conn_id</strong> (<em>str</em>) &#x2013; connection id from airflow Connections</li>
<li><strong>remote_host</strong> (<em>str</em>) &#x2013; remote host to connect</li>
<li><strong>local_filepath</strong> (<em>str</em>) &#x2013; local file path to get or put. (templated)</li>
<li><strong>remote_filepath</strong> (<em>str</em>) &#x2013; remote file path to get or put. (templated)</li>
<li><strong>operation</strong> &#x2013; specify operation &#x2018;get&#x2019; or &#x2018;put&#x2019;, defaults to get</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.contrib.operators.slack_webhook_operator.SlackWebhookOperator(http_conn_id=None, webhook_token=None, message=&apos;&apos;, channel=None, username=None, icon_emoji=None, link_names=False, proxy=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.operators.http_operator.SimpleHttpOperator" title="airflow.operators.http_operator.SimpleHttpOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.operators.http_operator.SimpleHttpOperator</span></code></a></p>
<p>This operator allows you to post messages to Slack using incoming webhooks.
Takes both Slack webhook token directly and connection that has Slack webhook token.
If both supplied, Slack webhook token will be used.</p>
<p>Each Slack webhook token can be pre-configured to use a specific channel, username and
icon. You can override these defaults in this hook.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>conn_id</strong> (<em>str</em>) &#x2013; connection that has Slack webhook token in the extra field</li>
<li><strong>webhook_token</strong> (<em>str</em>) &#x2013; Slack webhook token</li>
<li><strong>message</strong> (<em>str</em>) &#x2013; The message you want to send on Slack</li>
<li><strong>channel</strong> (<em>str</em>) &#x2013; The channel the message should be posted to</li>
<li><strong>username</strong> (<em>str</em>) &#x2013; The username to post to slack with</li>
<li><strong>icon_emoji</strong> (<em>str</em>) &#x2013; The emoji to use as icon for the user posting to Slack</li>
<li><strong>link_names</strong> (<em>bool</em>) &#x2013; Whether or not to find and link channel and usernames in your
message</li>
<li><strong>proxy</strong> (<em>str</em>) &#x2013; Proxy to use to make the Slack webhook call</li>
</ul>
</td>
</tr>
</tbody>
</table>

<pre>
execute(context)</pre>
<p>Call the SparkSqlHook to run the provided sql query</p>







<pre>
class airflow.contrib.operators.snowflake_operator.SnowflakeOperator(sql, snowflake_conn_id=&apos;snowflake_default&apos;, parameters=None, autocommit=True, warehouse=None, database=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Executes sql code in a Snowflake database</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>snowflake_conn_id</strong> (<em>string</em>) &#x2013; reference to specific snowflake connection id</li>
<li><strong>sql</strong> (<em>Can receive a str representing a sql statement</em><em>,
</em><em>a list of str</em><em> (</em><em>sql statements</em><em>)</em><em>, or </em><em>reference to a template file.
Template reference are recognized by str ending in &apos;.sql&apos;</em>) &#x2013; the sql code to be executed. (templated)</li>
<li><strong>warehouse</strong> (<em>string</em>) &#x2013; name of warehouse which overwrite defined
one in connection</li>
<li><strong>database</strong> (<em>string</em>) &#x2013; name of database which overwrite defined one in connection</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.contrib.operators.spark_jdbc_operator.SparkJDBCOperator(spark_app_name=&apos;airflow-spark-jdbc&apos;, spark_conn_id=&apos;spark-default&apos;, spark_conf=None, spark_py_files=None, spark_files=None, spark_jars=None, num_executors=None, executor_cores=None, executor_memory=None, driver_memory=None, verbose=False, keytab=None, principal=None, cmd_type=&apos;spark_to_jdbc&apos;, jdbc_table=None, jdbc_conn_id=&apos;jdbc-default&apos;, jdbc_driver=None, metastore_table=None, jdbc_truncate=False, save_mode=None, save_format=None, batch_size=None, fetch_size=None, num_partitions=None, partition_column=None, lower_bound=None, upper_bound=None, create_table_column_types=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.contrib.operators.spark_submit_operator.SparkSubmitOperator" title="airflow.contrib.operators.spark_submit_operator.SparkSubmitOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.contrib.operators.spark_submit_operator.SparkSubmitOperator</span></code></a></p>
<p>This operator extends the SparkSubmitOperator specifically for performing data
transfers to/from JDBC-based databases with Apache Spark. As with the
SparkSubmitOperator, it assumes that the &#x201C;spark-submit&#x201D; binary is available on the
PATH.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first simple">
<li><strong>spark_app_name</strong> (<em>str</em>) &#x2013; Name of the job (default airflow-spark-jdbc)</li>
<li><strong>spark_conn_id</strong> (<em>str</em>) &#x2013; Connection id as configured in Airflow administration</li>
<li><strong>spark_conf</strong> (<em>dict</em>) &#x2013; Any additional Spark configuration properties</li>
<li><strong>spark_py_files</strong> (<em>str</em>) &#x2013; Additional python files used (.zip, .egg, or .py)</li>
<li><strong>spark_files</strong> (<em>str</em>) &#x2013; Additional files to upload to the container running the job</li>
<li><strong>spark_jars</strong> (<em>str</em>) &#x2013; Additional jars to upload and add to the driver and
executor classpath</li>
<li><strong>num_executors</strong> (<em>int</em>) &#x2013; number of executor to run. This should be set so as to manage
the number of connections made with the JDBC database</li>
<li><strong>executor_cores</strong> (<em>int</em>) &#x2013; Number of cores per executor</li>
<li><strong>executor_memory</strong> (<em>str</em>) &#x2013; Memory per executor (e.g. 1000M, 2G)</li>
<li><strong>driver_memory</strong> (<em>str</em>) &#x2013; Memory allocated to the driver (e.g. 1000M, 2G)</li>
<li><strong>verbose</strong> (<em>bool</em>) &#x2013; Whether to pass the verbose flag to spark-submit for debugging</li>
<li><strong>keytab</strong> (<em>str</em>) &#x2013; Full path to the file that contains the keytab</li>
<li><strong>principal</strong> (<em>str</em>) &#x2013; The name of the kerberos principal used for keytab</li>
<li><strong>cmd_type</strong> (<em>str</em>) &#x2013; Which way the data should flow. 2 possible values:
spark_to_jdbc: data written by spark from metastore to jdbc
jdbc_to_spark: data written by spark from jdbc to metastore</li>
<li><strong>jdbc_table</strong> (<em>str</em>) &#x2013; The name of the JDBC table</li>
<li><strong>jdbc_conn_id</strong> &#x2013; Connection id used for connection to JDBC database</li>
<li><strong>jdbc_driver</strong> (<em>str</em>) &#x2013; Name of the JDBC driver to use for the JDBC connection. This
driver (usually a jar) should be passed in the &#x2018;jars&#x2019; parameter</li>
<li><strong>metastore_table</strong> (<em>str</em>) &#x2013; The name of the metastore table,</li>
<li><strong>jdbc_truncate</strong> (<em>bool</em>) &#x2013; (spark_to_jdbc only) Whether or not Spark should truncate or
drop and recreate the JDBC table. This only takes effect if
&#x2018;save_mode&#x2019; is set to Overwrite. Also, if the schema is
different, Spark cannot truncate, and will drop and recreate</li>
<li><strong>save_mode</strong> (<em>str</em>) &#x2013; The Spark save-mode to use (e.g. overwrite, append, etc.)</li>
<li><strong>save_format</strong> (<em>str</em>) &#x2013; (jdbc_to_spark-only) The Spark save-format to use (e.g. parquet)</li>
<li><strong>batch_size</strong> (<em>int</em>) &#x2013; (spark_to_jdbc only) The size of the batch to insert per round
trip to the JDBC database. Defaults to 1000</li>
<li><strong>fetch_size</strong> (<em>int</em>) &#x2013; (jdbc_to_spark only) The size of the batch to fetch per round trip
from the JDBC database. Default depends on the JDBC driver</li>
<li><strong>num_partitions</strong> (<em>int</em>) &#x2013; The maximum number of partitions that can be used by Spark
simultaneously, both for spark_to_jdbc and jdbc_to_spark
operations. This will also cap the number of JDBC connections
that can be opened</li>
<li><strong>partition_column</strong> (<em>str</em>) &#x2013; (jdbc_to_spark-only) A numeric column to be used to
partition the metastore table by. If specified, you must
also specify:
num_partitions, lower_bound, upper_bound</li>
<li><strong>lower_bound</strong> (<em>int</em>) &#x2013; (jdbc_to_spark-only) Lower bound of the range of the numeric
partition column to fetch. If specified, you must also specify:
num_partitions, partition_column, upper_bound</li>
<li><strong>upper_bound</strong> (<em>int</em>) &#x2013; (jdbc_to_spark-only) Upper bound of the range of the numeric
partition column to fetch. If specified, you must also specify:
num_partitions, partition_column, lower_bound</li>
<li><strong>create_table_column_types</strong> &#x2013; (spark_to_jdbc-only) The database column data types
to use instead of the defaults, when creating the
table. Data type information should be specified in
the same format as CREATE TABLE columns syntax
(e.g: &#x201C;name CHAR(64), comments VARCHAR(1024)&#x201D;).
The specified types should be valid spark sql data
types.</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Type:</th>
<td class="field-body"><p class="first last">jdbc_conn_id: str</p>
</td>
</tr>
</tbody>
</table>

<pre>
execute(context)</pre>
<p>Call the SparkSubmitHook to run the provided spark job</p>







<pre>
class airflow.contrib.operators.spark_sql_operator.SparkSqlOperator(sql, conf=None, conn_id=&apos;spark_sql_default&apos;, total_executor_cores=None, executor_cores=None, executor_memory=None, keytab=None, principal=None, master=&apos;yarn&apos;, name=&apos;default-name&apos;, num_executors=None, yarn_queue=&apos;default&apos;, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Execute Spark SQL query</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>sql</strong> (<em>str</em>) &#x2013; The SQL query to execute. (templated)</li>
<li><strong>conf</strong> (<em>str</em><em> (</em><em>format: PROP=VALUE</em><em>)</em>) &#x2013; arbitrary Spark configuration property</li>
<li><strong>conn_id</strong> (<em>str</em>) &#x2013; connection_id string</li>
<li><strong>total_executor_cores</strong> (<em>int</em>) &#x2013; (Standalone &amp; Mesos only) Total cores for all
executors (Default: all the available cores on the worker)</li>
<li><strong>executor_cores</strong> (<em>int</em>) &#x2013; (Standalone &amp; YARN only) Number of cores per
executor (Default: 2)</li>
<li><strong>executor_memory</strong> (<em>str</em>) &#x2013; Memory per executor (e.g. 1000M, 2G) (Default: 1G)</li>
<li><strong>keytab</strong> (<em>str</em>) &#x2013; Full path to the file that contains the keytab</li>
<li><strong>master</strong> (<em>str</em>) &#x2013; spark://host:port, mesos://host:port, yarn, or local</li>
<li><strong>name</strong> (<em>str</em>) &#x2013; Name of the job</li>
<li><strong>num_executors</strong> (<em>int</em>) &#x2013; Number of executors to launch</li>
<li><strong>verbose</strong> (<em>bool</em>) &#x2013; Whether to pass the verbose flag to spark-sql</li>
<li><strong>yarn_queue</strong> (<em>str</em>) &#x2013; The YARN queue to submit to (Default: &#x201C;default&#x201D;)</li>
</ul>
</td>
</tr>
</tbody>
</table>

<pre>
execute(context)</pre>
<p>Call the SparkSqlHook to run the provided sql query</p>







<pre>
class airflow.contrib.operators.spark_submit_operator.SparkSubmitOperator(application=&apos;&apos;, conf=None, conn_id=&apos;spark_default&apos;, files=None, py_files=None, driver_classpath=None, jars=None, java_class=None, packages=None, exclude_packages=None, repositories=None, total_executor_cores=None, executor_cores=None, executor_memory=None, driver_memory=None, keytab=None, principal=None, name=&apos;airflow-spark&apos;, num_executors=None, application_args=None, env_vars=None, verbose=False, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>This hook is a wrapper around the spark-submit binary to kick off a spark-submit job.
It requires that the &#x201C;spark-submit&#x201D; binary is in the PATH or the spark-home is set
in the extra on the connection.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>application</strong> (<em>str</em>) &#x2013; The application that submitted as a job, either jar or
py file. (templated)</li>
<li><strong>conf</strong> (<em>dict</em>) &#x2013; Arbitrary Spark configuration properties</li>
<li><strong>conn_id</strong> (<em>str</em>) &#x2013; The connection id as configured in Airflow administration. When an
invalid connection_id is supplied, it will default to yarn.</li>
<li><strong>files</strong> (<em>str</em>) &#x2013; Upload additional files to the executor running the job, separated by a
comma. Files will be placed in the working directory of each executor.
For example, serialized objects.</li>
<li><strong>py_files</strong> (<em>str</em>) &#x2013; Additional python files used by the job, can be .zip, .egg or .py.</li>
<li><strong>jars</strong> (<em>str</em>) &#x2013; Submit additional jars to upload and place them in executor classpath.</li>
<li><strong>driver_classpath</strong> (<em>str</em>) &#x2013; Additional, driver-specific, classpath settings.</li>
<li><strong>java_class</strong> (<em>str</em>) &#x2013; the main class of the Java application</li>
<li><strong>packages</strong> (<em>str</em>) &#x2013; Comma-separated list of maven coordinates of jars to include on the
driver and executor classpaths. (templated)</li>
<li><strong>exclude_packages</strong> (<em>str</em>) &#x2013; Comma-separated list of maven coordinates of jars to exclude
while resolving the dependencies provided in &#x2018;packages&#x2019;</li>
<li><strong>repositories</strong> (<em>str</em>) &#x2013; Comma-separated list of additional remote repositories to search
for the maven coordinates given with &#x2018;packages&#x2019;</li>
<li><strong>total_executor_cores</strong> (<em>int</em>) &#x2013; (Standalone &amp; Mesos only) Total cores for all executors
(Default: all the available cores on the worker)</li>
<li><strong>executor_cores</strong> (<em>int</em>) &#x2013; (Standalone &amp; YARN only) Number of cores per executor
(Default: 2)</li>
<li><strong>executor_memory</strong> (<em>str</em>) &#x2013; Memory per executor (e.g. 1000M, 2G) (Default: 1G)</li>
<li><strong>driver_memory</strong> (<em>str</em>) &#x2013; Memory allocated to the driver (e.g. 1000M, 2G) (Default: 1G)</li>
<li><strong>keytab</strong> (<em>str</em>) &#x2013; Full path to the file that contains the keytab</li>
<li><strong>principal</strong> (<em>str</em>) &#x2013; The name of the kerberos principal used for keytab</li>
<li><strong>name</strong> (<em>str</em>) &#x2013; Name of the job (default airflow-spark). (templated)</li>
<li><strong>num_executors</strong> (<em>int</em>) &#x2013; Number of executors to launch</li>
<li><strong>application_args</strong> (<em>list</em>) &#x2013; Arguments for the application being submitted</li>
<li><strong>env_vars</strong> (<em>dict</em>) &#x2013; Environment variables for spark-submit. It
supports yarn and k8s mode too.</li>
<li><strong>verbose</strong> (<em>bool</em>) &#x2013; Whether to pass the verbose flag to spark-submit process for debugging</li>
</ul>
</td>
</tr>
</tbody>
</table>

<pre>
execute(context)</pre>
<p>Call the SparkSubmitHook to run the provided spark job</p>







<pre>
class airflow.contrib.operators.sqoop_operator.SqoopOperator(conn_id=&apos;sqoop_default&apos;, cmd_type=&apos;import&apos;, table=None, query=None, target_dir=None, append=None, file_type=&apos;text&apos;, columns=None, num_mappers=None, split_by=None, where=None, export_dir=None, input_null_string=None, input_null_non_string=None, staging_table=None, clear_staging_table=False, enclosed_by=None, escaped_by=None, input_fields_terminated_by=None, input_lines_terminated_by=None, input_optionally_enclosed_by=None, batch=False, direct=False, driver=None, verbose=False, relaxed_isolation=False, properties=None, hcatalog_database=None, hcatalog_table=None, create_hcatalog_table=False, extra_import_options=None, extra_export_options=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Execute a Sqoop job.
Documentation for Apache Sqoop can be found here:</p>
<blockquote>
<div><a class="reference external" href="https://sqoop.apache.org/docs/1.4.2/SqoopUserGuide.html">https://sqoop.apache.org/docs/1.4.2/SqoopUserGuide.html</a>.</div>
</blockquote>

<pre>
execute(context)</pre>
<p>Execute sqoop job</p>







<pre>
class airflow.contrib.operators.ssh_operator.SSHOperator(ssh_hook=None, ssh_conn_id=None, remote_host=None, command=None, timeout=10, do_xcom_push=False, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>SSHOperator to execute commands on given remote host using the ssh_hook.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>ssh_hook</strong> (<code class="xref py py-class docutils literal notranslate"><span class="pre">SSHHook</span></code>) &#x2013; predefined ssh_hook to use for remote execution</li>
<li><strong>ssh_conn_id</strong> (<em>str</em>) &#x2013; connection id from airflow Connections</li>
<li><strong>remote_host</strong> (<em>str</em>) &#x2013; remote host to connect</li>
<li><strong>command</strong> (<em>str</em>) &#x2013; command to execute on remote host. (templated)</li>
<li><strong>timeout</strong> (<em>int</em>) &#x2013; timeout (in seconds) for executing the command.</li>
<li><strong>do_xcom_push</strong> (<em>bool</em>) &#x2013; return the stdout which also get set in xcom by airflow platform</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.contrib.operators.vertica_operator.VerticaOperator(sql, vertica_conn_id=&apos;vertica_default&apos;, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Executes sql code in a specific Vertica database</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>vertica_conn_id</strong> (<em>string</em>) &#x2013; reference to a specific Vertica database</li>
<li><strong>sql</strong> (<em>Can receive a str representing a sql statement</em><em>,
</em><em>a list of str</em><em> (</em><em>sql statements</em><em>)</em><em>, or </em><em>reference to a template file.
Template reference are recognized by str ending in &apos;.sql&apos;</em>) &#x2013; the sql code to be executed. (templated)</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.contrib.operators.vertica_to_hive.VerticaToHiveTransfer(sql, hive_table, create=True, recreate=False, partition=None, delimiter=u&apos;x01&apos;, vertica_conn_id=&apos;vertica_default&apos;, hive_cli_conn_id=&apos;hive_cli_default&apos;, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Moves data from Vertia to Hive. The operator runs
your query against Vertia, stores the file locally
before loading it into a Hive table. If the <code class="docutils literal notranslate"><span class="pre">create</span></code> or
<code class="docutils literal notranslate"><span class="pre">recreate</span></code> arguments are set to <code class="docutils literal notranslate"><span class="pre">True</span></code>,
a <code class="docutils literal notranslate"><span class="pre">CREATE</span> <span class="pre">TABLE</span></code> and <code class="docutils literal notranslate"><span class="pre">DROP</span> <span class="pre">TABLE</span></code> statements are generated.
Hive data types are inferred from the cursor&#x2019;s metadata.
Note that the table generated in Hive uses <code class="docutils literal notranslate"><span class="pre">STORED</span> <span class="pre">AS</span> <span class="pre">textfile</span></code>
which isn&#x2019;t the most efficient serialization format. If a
large amount of data is loaded and/or if the table gets
queried considerably, you may want to use this operator only to
stage the data into a temporary table before loading it into its
final destination using a <code class="docutils literal notranslate"><span class="pre">HiveOperator</span></code>.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>sql</strong> (<em>str</em>) &#x2013; SQL query to execute against the Vertia database. (templated)</li>
<li><strong>hive_table</strong> (<em>str</em>) &#x2013; target Hive table, use dot notation to target a
specific database. (templated)</li>
<li><strong>create</strong> (<em>bool</em>) &#x2013; whether to create the table if it doesn&#x2019;t exist</li>
<li><strong>recreate</strong> (<em>bool</em>) &#x2013; whether to drop and recreate the table at every execution</li>
<li><strong>partition</strong> (<em>dict</em>) &#x2013; target partition as a dict of partition columns
and values. (templated)</li>
<li><strong>delimiter</strong> (<em>str</em>) &#x2013; field delimiter in the file</li>
<li><strong>vertica_conn_id</strong> (<em>str</em>) &#x2013; source Vertica connection</li>
<li><strong>hive_conn_id</strong> (<em>str</em>) &#x2013; destination hive connection</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.contrib.operators.winrm_operator.WinRMOperator(winrm_hook=None, ssh_conn_id=None, remote_host=None, command=None, timeout=10, do_xcom_push=False, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>WinRMOperator to execute commands on given remote host using the winrm_hook.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>winrm_hook</strong> (<code class="xref py py-class docutils literal notranslate"><span class="pre">WinRMHook</span></code>) &#x2013; predefined ssh_hook to use for remote execution</li>
<li><strong>ssh_conn_id</strong> (<em>str</em>) &#x2013; connection id from airflow Connections</li>
<li><strong>remote_host</strong> (<em>str</em>) &#x2013; remote host to connect</li>
<li><strong>command</strong> (<em>str</em>) &#x2013; command to execute on remote host. (templated)</li>
<li><strong>timeout</strong> (<em>int</em>) &#x2013; timeout for executing the command.</li>
<li><strong>do_xcom_push</strong> (<em>bool</em>) &#x2013; return the stdout which also get set in xcom by airflow platform</li>
</ul>
</td>
</tr>
</tbody>
</table>



</div>
<div class="section" id="id11">
<h4 class="sigil_not_in_toc">Sensors</h4>

<pre>
class airflow.contrib.sensors.aws_redshift_cluster_sensor.AwsRedshiftClusterSensor(cluster_identifier, target_status=&apos;available&apos;, aws_conn_id=&apos;aws_default&apos;, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.sensors.base_sensor_operator.BaseSensorOperator" title="airflow.sensors.base_sensor_operator.BaseSensorOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.sensors.base_sensor_operator.BaseSensorOperator</span></code></a></p>
<p>Waits for a Redshift cluster to reach a specific status.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>cluster_identifier</strong> (<em>str</em>) &#x2013; The identifier for the cluster being pinged.</li>
<li><strong>target_status</strong> (<em>str</em>) &#x2013; The cluster status desired.</li>
</ul>
</td>
</tr>
</tbody>
</table>

<pre>
poke(context)</pre>
<p>Function that the sensors defined while deriving this class should
override.</p>







<pre>
class airflow.contrib.sensors.bash_sensor.BashSensor(bash_command, env=None, output_encoding=&apos;utf-8&apos;, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.sensors.base_sensor_operator.BaseSensorOperator" title="airflow.sensors.base_sensor_operator.BaseSensorOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.sensors.base_sensor_operator.BaseSensorOperator</span></code></a></p>
<p>Executes a bash command/script and returns True if and only if the
return code is 0.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>bash_command</strong> (<em>string</em>) &#x2013; The command, set of commands or reference to a
bash script (must be &#x2018;.sh&#x2019;) to be executed.</li>
<li><strong>env</strong> (<em>dict</em>) &#x2013; If env is not None, it must be a mapping that defines the
environment variables for the new process; these are used instead
of inheriting the current process environment, which is the default
behavior. (templated)</li>
<li><strong>output_encoding</strong> (<em>string</em>) &#x2013; output encoding of bash command.</li>
</ul>
</td>
</tr>
</tbody>
</table>

<pre>
poke(context)</pre>
<p>Execute the bash command in a temporary directory
which will be cleaned afterwards</p>







<pre>
class airflow.contrib.sensors.bigquery_sensor.BigQueryTableSensor(project_id, dataset_id, table_id, bigquery_conn_id=&apos;bigquery_default_conn&apos;, delegate_to=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.sensors.base_sensor_operator.BaseSensorOperator" title="airflow.sensors.base_sensor_operator.BaseSensorOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.sensors.base_sensor_operator.BaseSensorOperator</span></code></a></p>
<p>Checks for the existence of a table in Google Bigquery.</p>
<blockquote>
<div><table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name" colspan="2">param project_id:</th>
</tr>
<tr class="field-odd field"><td>&#xA0;</td>
<td class="field-body">The Google cloud project in which to look for the table.
The connection supplied to the hook must provide
access to the specified project.</td>
</tr>
<tr class="field-even field"><th class="field-name" colspan="2">type project_id:</th>
</tr>
<tr class="field-even field"><td>&#xA0;</td>
<td class="field-body">string</td>
</tr>
<tr class="field-odd field"><th class="field-name" colspan="2">param dataset_id:</th>
</tr>
<tr class="field-odd field"><td>&#xA0;</td>
<td class="field-body">The name of the dataset in which to look for the table.
storage bucket.</td>
</tr>
<tr class="field-even field"><th class="field-name" colspan="2">type dataset_id:</th>
</tr>
<tr class="field-even field"><td>&#xA0;</td>
<td class="field-body">string</td>
</tr>
<tr class="field-odd field"><th class="field-name">param table_id:</th>
<td class="field-body">The name of the table to check the existence of.</td>
</tr>
<tr class="field-even field"><th class="field-name">type table_id:</th>
<td class="field-body">string</td>
</tr>
<tr class="field-odd field"><th class="field-name" colspan="2">param bigquery_conn_id:</th>
</tr>
<tr class="field-odd field"><td>&#xA0;</td>
<td class="field-body">The connection ID to use when connecting to
Google BigQuery.</td>
</tr>
<tr class="field-even field"><th class="field-name" colspan="2">type bigquery_conn_id:</th>
</tr>
<tr class="field-even field"><td>&#xA0;</td>
<td class="field-body">string</td>
</tr>
<tr class="field-odd field"><th class="field-name" colspan="2">param delegate_to:</th>
</tr>
<tr class="field-odd field"><td>&#xA0;</td>
<td class="field-body">The account to impersonate, if any.
For this to work, the service account making the request must
have domain-wide delegation enabled.</td>
</tr>
<tr class="field-even field"><th class="field-name" colspan="2">type delegate_to:</th>
</tr>
<tr class="field-even field"><td>&#xA0;</td>
<td class="field-body">string</td>
</tr>
</tbody>
</table>
</div>
</blockquote>

<pre>
poke(context)</pre>
<p>Function that the sensors defined while deriving this class should
override.</p>







<pre>
class airflow.contrib.sensors.datadog_sensor.DatadogSensor(datadog_conn_id=&apos;datadog_default&apos;, from_seconds_ago=3600, up_to_seconds_from_now=0, priority=None, sources=None, tags=None, response_check=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.sensors.base_sensor_operator.BaseSensorOperator" title="airflow.sensors.base_sensor_operator.BaseSensorOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.sensors.base_sensor_operator.BaseSensorOperator</span></code></a></p>
<p>A sensor to listen, with a filter, to datadog event streams and determine
if some event was emitted.</p>
<p>Depends on the datadog API, which has to be deployed on the same server where
Airflow runs.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>datadog_conn_id</strong> &#x2013; The connection to datadog, containing metadata for api keys.</li>
<li><strong>datadog_conn_id</strong> &#x2013; string</li>
</ul>
</td>
</tr>
</tbody>
</table>

<pre>
poke(context)</pre>
<p>Function that the sensors defined while deriving this class should
override.</p>







<pre>
class airflow.contrib.sensors.emr_base_sensor.EmrBaseSensor(aws_conn_id=&apos;aws_default&apos;, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.sensors.base_sensor_operator.BaseSensorOperator" title="airflow.sensors.base_sensor_operator.BaseSensorOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.sensors.base_sensor_operator.BaseSensorOperator</span></code></a></p>
<p>Contains general sensor behavior for EMR.
Subclasses should implement get_emr_response() and state_from_response() methods.
Subclasses should also implement NON_TERMINAL_STATES and FAILED_STATE constants.</p>

<pre>
poke(context)</pre>
<p>Function that the sensors defined while deriving this class should
override.</p>







<pre>
class airflow.contrib.sensors.emr_job_flow_sensor.EmrJobFlowSensor(job_flow_id, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.contrib.sensors.emr_base_sensor.EmrBaseSensor" title="airflow.contrib.sensors.emr_base_sensor.EmrBaseSensor"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.contrib.sensors.emr_base_sensor.EmrBaseSensor</span></code></a></p>
<p>Asks for the state of the JobFlow until it reaches a terminal state.
If it fails the sensor errors, failing the task.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><strong>job_flow_id</strong> (<em>string</em>) &#x2013; job_flow_id to check the state of</td>
</tr>
</tbody>
</table>




<pre>
class airflow.contrib.sensors.emr_step_sensor.EmrStepSensor(job_flow_id, step_id, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.contrib.sensors.emr_base_sensor.EmrBaseSensor" title="airflow.contrib.sensors.emr_base_sensor.EmrBaseSensor"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.contrib.sensors.emr_base_sensor.EmrBaseSensor</span></code></a></p>
<p>Asks for the state of the step until it reaches a terminal state.
If it fails the sensor errors, failing the task.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>job_flow_id</strong> (<em>string</em>) &#x2013; job_flow_id which contains the step check the state of</li>
<li><strong>step_id</strong> (<em>string</em>) &#x2013; step to check the state of</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.contrib.sensors.file_sensor.FileSensor(filepath, fs_conn_id=&apos;fs_default2&apos;, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.sensors.base_sensor_operator.BaseSensorOperator" title="airflow.sensors.base_sensor_operator.BaseSensorOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.sensors.base_sensor_operator.BaseSensorOperator</span></code></a></p>
<p>Waits for a file or folder to land in a filesystem.</p>
<p>If the path given is a directory then this sensor will only return true if
any files exist inside it (either directly, or within a subdirectory)</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>fs_conn_id</strong> (<em>string</em>) &#x2013; reference to the File (path)
connection id</li>
<li><strong>filepath</strong> &#x2013; File or folder name (relative to
the base path set within the connection)</li>
</ul>
</td>
</tr>
</tbody>
</table>

<pre>
poke(context)</pre>
<p>Function that the sensors defined while deriving this class should
override.</p>







<pre>
class airflow.contrib.sensors.ftp_sensor.FTPSensor(path, ftp_conn_id=&apos;ftp_default&apos;, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.sensors.base_sensor_operator.BaseSensorOperator" title="airflow.sensors.base_sensor_operator.BaseSensorOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.sensors.base_sensor_operator.BaseSensorOperator</span></code></a></p>
<p>Waits for a file or directory to be present on FTP.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>path</strong> (<em>str</em>) &#x2013; Remote file or directory path</li>
<li><strong>ftp_conn_id</strong> (<em>str</em>) &#x2013; The connection to run the sensor against</li>
</ul>
</td>
</tr>
</tbody>
</table>

<pre>
poke(context)</pre>
<p>Function that the sensors defined while deriving this class should
override.</p>







<pre>
class airflow.contrib.sensors.ftp_sensor.FTPSSensor(path, ftp_conn_id=&apos;ftp_default&apos;, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.contrib.sensors.ftp_sensor.FTPSensor" title="airflow.contrib.sensors.ftp_sensor.FTPSensor"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.contrib.sensors.ftp_sensor.FTPSensor</span></code></a></p>
<p>Waits for a file or directory to be present on FTP over SSL.</p>




<pre>
class airflow.contrib.sensors.gcs_sensor.GoogleCloudStorageObjectSensor(bucket, object, google_cloud_conn_id=&apos;google_cloud_default&apos;, delegate_to=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.sensors.base_sensor_operator.BaseSensorOperator" title="airflow.sensors.base_sensor_operator.BaseSensorOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.sensors.base_sensor_operator.BaseSensorOperator</span></code></a></p>
<p>Checks for the existence of a file in Google Cloud Storage.
Create a new GoogleCloudStorageObjectSensor.</p>
<blockquote>
<div><table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">param bucket:</th>
<td class="field-body">The Google cloud storage bucket where the object is.</td>
</tr>
<tr class="field-even field"><th class="field-name">type bucket:</th>
<td class="field-body">string</td>
</tr>
<tr class="field-odd field"><th class="field-name">param object:</th>
<td class="field-body">The name of the object to check in the Google cloud
storage bucket.</td>
</tr>
<tr class="field-even field"><th class="field-name">type object:</th>
<td class="field-body">string</td>
</tr>
<tr class="field-odd field"><th class="field-name" colspan="2">param google_cloud_storage_conn_id:</th>
</tr>
<tr class="field-odd field"><td>&#xA0;</td>
<td class="field-body">The connection ID to use when
connecting to Google cloud storage.</td>
</tr>
<tr class="field-even field"><th class="field-name" colspan="2">type google_cloud_storage_conn_id:</th>
</tr>
<tr class="field-even field"><td>&#xA0;</td>
<td class="field-body">string</td>
</tr>
<tr class="field-odd field"><th class="field-name" colspan="2">param delegate_to:</th>
</tr>
<tr class="field-odd field"><td>&#xA0;</td>
<td class="field-body">The account to impersonate, if any.
For this to work, the service account making the request must have
domain-wide delegation enabled.</td>
</tr>
<tr class="field-even field"><th class="field-name" colspan="2">type delegate_to:</th>
</tr>
<tr class="field-even field"><td>&#xA0;</td>
<td class="field-body">string</td>
</tr>
</tbody>
</table>
</div>
</blockquote>

<pre>
poke(context)</pre>
<p>Function that the sensors defined while deriving this class should
override.</p>







<pre>
class airflow.contrib.sensors.gcs_sensor.GoogleCloudStorageObjectUpdatedSensor(bucket, object, ts_func=&lt;function ts_function&gt;, google_cloud_conn_id=&apos;google_cloud_default&apos;, delegate_to=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.sensors.base_sensor_operator.BaseSensorOperator" title="airflow.sensors.base_sensor_operator.BaseSensorOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.sensors.base_sensor_operator.BaseSensorOperator</span></code></a></p>
<p>Checks if an object is updated in Google Cloud Storage.
Create a new GoogleCloudStorageObjectUpdatedSensor.</p>
<blockquote>
<div><table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">param bucket:</th>
<td class="field-body">The Google cloud storage bucket where the object is.</td>
</tr>
<tr class="field-even field"><th class="field-name">type bucket:</th>
<td class="field-body">string</td>
</tr>
<tr class="field-odd field"><th class="field-name">param object:</th>
<td class="field-body">The name of the object to download in the Google cloud
storage bucket.</td>
</tr>
<tr class="field-even field"><th class="field-name">type object:</th>
<td class="field-body">string</td>
</tr>
<tr class="field-odd field"><th class="field-name">param ts_func:</th>
<td class="field-body">Callback for defining the update condition. The default callback
returns execution_date + schedule_interval. The callback takes the context
as parameter.</td>
</tr>
<tr class="field-even field"><th class="field-name">type ts_func:</th>
<td class="field-body">function</td>
</tr>
<tr class="field-odd field"><th class="field-name" colspan="2">param google_cloud_storage_conn_id:</th>
</tr>
<tr class="field-odd field"><td>&#xA0;</td>
<td class="field-body">The connection ID to use when
connecting to Google cloud storage.</td>
</tr>
<tr class="field-even field"><th class="field-name" colspan="2">type google_cloud_storage_conn_id:</th>
</tr>
<tr class="field-even field"><td>&#xA0;</td>
<td class="field-body">string</td>
</tr>
<tr class="field-odd field"><th class="field-name" colspan="2">param delegate_to:</th>
</tr>
<tr class="field-odd field"><td>&#xA0;</td>
<td class="field-body">The account to impersonate, if any.
For this to work, the service account making the request must have domain-wide
delegation enabled.</td>
</tr>
<tr class="field-even field"><th class="field-name" colspan="2">type delegate_to:</th>
</tr>
<tr class="field-even field"><td>&#xA0;</td>
<td class="field-body">string</td>
</tr>
</tbody>
</table>
</div>
</blockquote>

<pre>
poke(context)</pre>
<p>Function that the sensors defined while deriving this class should
override.</p>







<pre>
class airflow.contrib.sensors.gcs_sensor.GoogleCloudStoragePrefixSensor(bucket, prefix, google_cloud_conn_id=&apos;google_cloud_default&apos;, delegate_to=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.sensors.base_sensor_operator.BaseSensorOperator" title="airflow.sensors.base_sensor_operator.BaseSensorOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.sensors.base_sensor_operator.BaseSensorOperator</span></code></a></p>
<p>Checks for the existence of a files at prefix in Google Cloud Storage bucket.
Create a new GoogleCloudStorageObjectSensor.</p>
<blockquote>
<div><table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">param bucket:</th>
<td class="field-body">The Google cloud storage bucket where the object is.</td>
</tr>
<tr class="field-even field"><th class="field-name">type bucket:</th>
<td class="field-body">string</td>
</tr>
<tr class="field-odd field"><th class="field-name">param prefix:</th>
<td class="field-body">The name of the prefix to check in the Google cloud
storage bucket.</td>
</tr>
<tr class="field-even field"><th class="field-name">type prefix:</th>
<td class="field-body">string</td>
</tr>
<tr class="field-odd field"><th class="field-name" colspan="2">param google_cloud_storage_conn_id:</th>
</tr>
<tr class="field-odd field"><td>&#xA0;</td>
<td class="field-body">The connection ID to use when
connecting to Google cloud storage.</td>
</tr>
<tr class="field-even field"><th class="field-name" colspan="2">type google_cloud_storage_conn_id:</th>
</tr>
<tr class="field-even field"><td>&#xA0;</td>
<td class="field-body">string</td>
</tr>
<tr class="field-odd field"><th class="field-name" colspan="2">param delegate_to:</th>
</tr>
<tr class="field-odd field"><td>&#xA0;</td>
<td class="field-body">The account to impersonate, if any.
For this to work, the service account making the request must have
domain-wide delegation enabled.</td>
</tr>
<tr class="field-even field"><th class="field-name" colspan="2">type delegate_to:</th>
</tr>
<tr class="field-even field"><td>&#xA0;</td>
<td class="field-body">string</td>
</tr>
</tbody>
</table>
</div>
</blockquote>

<pre>
poke(context)</pre>
<p>Function that the sensors defined while deriving this class should
override.</p>







<pre>
class airflow.contrib.sensors.hdfs_sensor.HdfsSensorFolder(be_empty=False, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.sensors.hdfs_sensor.HdfsSensor" title="airflow.sensors.hdfs_sensor.HdfsSensor"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.sensors.hdfs_sensor.HdfsSensor</span></code></a></p>

<pre>
poke(context)</pre>
<p>poke for a non empty directory</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Returns:</th>
<td class="field-body">Bool depending on the search criteria</td>
</tr>
</tbody>
</table>







<pre>
class airflow.contrib.sensors.hdfs_sensor.HdfsSensorRegex(regex, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.sensors.hdfs_sensor.HdfsSensor" title="airflow.sensors.hdfs_sensor.HdfsSensor"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.sensors.hdfs_sensor.HdfsSensor</span></code></a></p>

<pre>
poke(context)</pre>
<p>poke matching files in a directory with self.regex</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Returns:</th>
<td class="field-body">Bool depending on the search criteria</td>
</tr>
</tbody>
</table>







<pre>
class airflow.contrib.sensors.jira_sensor.JiraSensor(jira_conn_id=&apos;jira_default&apos;, method_name=None, method_params=None, result_processor=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.sensors.base_sensor_operator.BaseSensorOperator" title="airflow.sensors.base_sensor_operator.BaseSensorOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.sensors.base_sensor_operator.BaseSensorOperator</span></code></a></p>
<p>Monitors a jira ticket for any change.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>jira_conn_id</strong> (<em>str</em>) &#x2013; reference to a pre-defined Jira Connection</li>
<li><strong>method_name</strong> (<em>str</em>) &#x2013; method name from jira-python-sdk to be execute</li>
<li><strong>method_params</strong> (<em>dict</em>) &#x2013; parameters for the method method_name</li>
<li><strong>result_processor</strong> (<em>function</em>) &#x2013; function that return boolean and act as a sensor response</li>
</ul>
</td>
</tr>
</tbody>
</table>

<pre>
poke(context)</pre>
<p>Function that the sensors defined while deriving this class should
override.</p>







<pre>
class airflow.contrib.sensors.pubsub_sensor.PubSubPullSensor(project, subscription, max_messages=5, return_immediately=False, ack_messages=False, gcp_conn_id=&apos;google_cloud_default&apos;, delegate_to=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.sensors.base_sensor_operator.BaseSensorOperator" title="airflow.sensors.base_sensor_operator.BaseSensorOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.sensors.base_sensor_operator.BaseSensorOperator</span></code></a></p>
<p>Pulls messages from a PubSub subscription and passes them through XCom.</p>
<p>This sensor operator will pull up to <code class="docutils literal notranslate"><span class="pre">max_messages</span></code> messages from the
specified PubSub subscription. When the subscription returns messages,
the poke method&#x2019;s criteria will be fulfilled and the messages will be
returned from the operator and passed through XCom for downstream tasks.</p>
<p>If <code class="docutils literal notranslate"><span class="pre">ack_messages</span></code> is set to True, messages will be immediately
acknowledged before being returned, otherwise, downstream tasks will be
responsible for acknowledging them.</p>
<p><code class="docutils literal notranslate"><span class="pre">project</span></code> and <code class="docutils literal notranslate"><span class="pre">subscription</span></code> are templated so you can use
variables in them.</p>

<pre>
execute(context)</pre>
<p>Overridden to allow messages to be passed</p>




<pre>
poke(context)</pre>
<p>Function that the sensors defined while deriving this class should
override.</p>







<pre>
class airflow.contrib.sensors.qubole_sensor.QuboleSensor(data, qubole_conn_id=&apos;qubole_default&apos;, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.sensors.base_sensor_operator.BaseSensorOperator" title="airflow.sensors.base_sensor_operator.BaseSensorOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.sensors.base_sensor_operator.BaseSensorOperator</span></code></a></p>
<p>Base class for all Qubole Sensors</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>qubole_conn_id</strong> (<em>string</em>) &#x2013; The qubole connection to run the sensor against</li>
<li><strong>data</strong> (<em>a JSON object</em>) &#x2013; a JSON object containing payload, whose presence needs to be checked</li>
</ul>
</td>
</tr>
</tbody>
</table>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">Both <code class="docutils literal notranslate"><span class="pre">data</span></code> and <code class="docutils literal notranslate"><span class="pre">qubole_conn_id</span></code> fields are template-supported. You can</p>
</div>
<p>also use <code class="docutils literal notranslate"><span class="pre">.txt</span></code> files for template driven use cases.</p>

<pre>
poke(context)</pre>
<p>Function that the sensors defined while deriving this class should
override.</p>







<pre>
class airflow.contrib.sensors.redis_key_sensor.RedisKeySensor(key, redis_conn_id, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.sensors.base_sensor_operator.BaseSensorOperator" title="airflow.sensors.base_sensor_operator.BaseSensorOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.sensors.base_sensor_operator.BaseSensorOperator</span></code></a></p>
<p>Checks for the existence of a key in a Redis database</p>

<pre>
poke(context)</pre>
<p>Function that the sensors defined while deriving this class should
override.</p>







<pre>
class airflow.contrib.sensors.sftp_sensor.SFTPSensor(path, sftp_conn_id=&apos;sftp_default&apos;, *args, **kwargs)</pre>
<p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.operators.sensors.BaseSensorOperator</span></code></p>
<p>Waits for a file or directory to be present on SFTP.
:param path: Remote file or directory path
:type path: str
:param sftp_conn_id: The connection to run the sensor against
:type sftp_conn_id: str</p>

<pre>
poke(context)</pre>
<p>Function that the sensors defined while deriving this class should
override.</p>







<pre>
class airflow.contrib.sensors.wasb_sensor.WasbBlobSensor(container_name, blob_name, wasb_conn_id=&apos;wasb_default&apos;, check_options=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.sensors.base_sensor_operator.BaseSensorOperator" title="airflow.sensors.base_sensor_operator.BaseSensorOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.sensors.base_sensor_operator.BaseSensorOperator</span></code></a></p>
<p>Waits for a blob to arrive on Azure Blob Storage.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>container_name</strong> (<em>str</em>) &#x2013; Name of the container.</li>
<li><strong>blob_name</strong> (<em>str</em>) &#x2013; Name of the blob.</li>
<li><strong>wasb_conn_id</strong> (<em>str</em>) &#x2013; Reference to the wasb connection.</li>
<li><strong>check_options</strong> (<em>dict</em>) &#x2013; Optional keyword arguments that
<cite>WasbHook.check_for_blob()</cite> takes.</li>
</ul>
</td>
</tr>
</tbody>
</table>

<pre>
poke(context)</pre>
<p>Function that the sensors defined while deriving this class should
override.</p>






</div>
</div>
</div>
<div class="section" id="macros">
<span id="id12"></span><h2 class="sigil_not_in_toc">Macros</h2>
<p>Here&#x2019;s a list of variables and macros that can be used in templates</p>
<div class="section" id="default-variables">
<h3 class="sigil_not_in_toc">Default Variables</h3>
<p>The Airflow engine passes a few variables by default that are accessible
in all templates</p>
<table border="1" class="docutils">
<colgroup>
<col width="30%">
<col width="70%">
</colgroup>
<thead valign="bottom">
<tr class="row-odd"><th class="head">Variable</th>
<th class="head">Description</th>
</tr>
</thead>
<tbody valign="top">
<tr class="row-even"><td><code class="docutils literal notranslate"><span class="pre">{{</span> <span class="pre">ds</span> <span class="pre">}}</span></code></td>
<td>the execution date as <code class="docutils literal notranslate"><span class="pre">YYYY-MM-DD</span></code></td>
</tr>
<tr class="row-odd"><td><code class="docutils literal notranslate"><span class="pre">{{</span> <span class="pre">ds_nodash</span> <span class="pre">}}</span></code></td>
<td>the execution date as <code class="docutils literal notranslate"><span class="pre">YYYYMMDD</span></code></td>
</tr>
<tr class="row-even"><td><code class="docutils literal notranslate"><span class="pre">{{</span> <span class="pre">prev_ds</span> <span class="pre">}}</span></code></td>
<td>the previous execution date as <code class="docutils literal notranslate"><span class="pre">YYYY-MM-DD</span></code>.
if <code class="docutils literal notranslate"><span class="pre">{{</span> <span class="pre">ds</span> <span class="pre">}}</span></code> is <code class="docutils literal notranslate"><span class="pre">2016-01-08</span></code> and <code class="docutils literal notranslate"><span class="pre">schedule_interval</span></code> is <code class="docutils literal notranslate"><span class="pre">@weekly</span></code>,
<code class="docutils literal notranslate"><span class="pre">{{</span> <span class="pre">prev_ds</span> <span class="pre">}}</span></code> will be <code class="docutils literal notranslate"><span class="pre">2016-01-01</span></code>.</td>
</tr>
<tr class="row-odd"><td><code class="docutils literal notranslate"><span class="pre">{{</span> <span class="pre">next_ds</span> <span class="pre">}}</span></code></td>
<td>the next execution date as <code class="docutils literal notranslate"><span class="pre">YYYY-MM-DD</span></code>.
if <code class="docutils literal notranslate"><span class="pre">{{</span> <span class="pre">ds</span> <span class="pre">}}</span></code> is <code class="docutils literal notranslate"><span class="pre">2016-01-01</span></code> and <code class="docutils literal notranslate"><span class="pre">schedule_interval</span></code> is <code class="docutils literal notranslate"><span class="pre">@weekly</span></code>,
<code class="docutils literal notranslate"><span class="pre">{{</span> <span class="pre">prev_ds</span> <span class="pre">}}</span></code> will be <code class="docutils literal notranslate"><span class="pre">2016-01-08</span></code>.</td>
</tr>
<tr class="row-even"><td><code class="docutils literal notranslate"><span class="pre">{{</span> <span class="pre">yesterday_ds</span> <span class="pre">}}</span></code></td>
<td>yesterday&#x2019;s date as <code class="docutils literal notranslate"><span class="pre">YYYY-MM-DD</span></code></td>
</tr>
<tr class="row-odd"><td><code class="docutils literal notranslate"><span class="pre">{{</span> <span class="pre">yesterday_ds_nodash</span> <span class="pre">}}</span></code></td>
<td>yesterday&#x2019;s date as <code class="docutils literal notranslate"><span class="pre">YYYYMMDD</span></code></td>
</tr>
<tr class="row-even"><td><code class="docutils literal notranslate"><span class="pre">{{</span> <span class="pre">tomorrow_ds</span> <span class="pre">}}</span></code></td>
<td>tomorrow&#x2019;s date as <code class="docutils literal notranslate"><span class="pre">YYYY-MM-DD</span></code></td>
</tr>
<tr class="row-odd"><td><code class="docutils literal notranslate"><span class="pre">{{</span> <span class="pre">tomorrow_ds_nodash</span> <span class="pre">}}</span></code></td>
<td>tomorrow&#x2019;s date as <code class="docutils literal notranslate"><span class="pre">YYYYMMDD</span></code></td>
</tr>
<tr class="row-even"><td><code class="docutils literal notranslate"><span class="pre">{{</span> <span class="pre">ts</span> <span class="pre">}}</span></code></td>
<td>same as <code class="docutils literal notranslate"><span class="pre">execution_date.isoformat()</span></code></td>
</tr>
<tr class="row-odd"><td><code class="docutils literal notranslate"><span class="pre">{{</span> <span class="pre">ts_nodash</span> <span class="pre">}}</span></code></td>
<td>same as <code class="docutils literal notranslate"><span class="pre">ts</span></code> without <code class="docutils literal notranslate"><span class="pre">-</span></code> and <code class="docutils literal notranslate"><span class="pre">:</span></code></td>
</tr>
<tr class="row-even"><td><code class="docutils literal notranslate"><span class="pre">{{</span> <span class="pre">execution_date</span> <span class="pre">}}</span></code></td>
<td>the execution_date, (datetime.datetime)</td>
</tr>
<tr class="row-odd"><td><code class="docutils literal notranslate"><span class="pre">{{</span> <span class="pre">prev_execution_date</span> <span class="pre">}}</span></code></td>
<td>the previous execution date (if available) (datetime.datetime)</td>
</tr>
<tr class="row-even"><td><code class="docutils literal notranslate"><span class="pre">{{</span> <span class="pre">next_execution_date</span> <span class="pre">}}</span></code></td>
<td>the next execution date (datetime.datetime)</td>
</tr>
<tr class="row-odd"><td><code class="docutils literal notranslate"><span class="pre">{{</span> <span class="pre">dag</span> <span class="pre">}}</span></code></td>
<td>the DAG object</td>
</tr>
<tr class="row-even"><td><code class="docutils literal notranslate"><span class="pre">{{</span> <span class="pre">task</span> <span class="pre">}}</span></code></td>
<td>the Task object</td>
</tr>
<tr class="row-odd"><td><code class="docutils literal notranslate"><span class="pre">{{</span> <span class="pre">macros</span> <span class="pre">}}</span></code></td>
<td>a reference to the macros package, described below</td>
</tr>
<tr class="row-even"><td><code class="docutils literal notranslate"><span class="pre">{{</span> <span class="pre">task_instance</span> <span class="pre">}}</span></code></td>
<td>the task_instance object</td>
</tr>
<tr class="row-odd"><td><code class="docutils literal notranslate"><span class="pre">{{</span> <span class="pre">end_date</span> <span class="pre">}}</span></code></td>
<td>same as <code class="docutils literal notranslate"><span class="pre">{{</span> <span class="pre">ds</span> <span class="pre">}}</span></code></td>
</tr>
<tr class="row-even"><td><code class="docutils literal notranslate"><span class="pre">{{</span> <span class="pre">latest_date</span> <span class="pre">}}</span></code></td>
<td>same as <code class="docutils literal notranslate"><span class="pre">{{</span> <span class="pre">ds</span> <span class="pre">}}</span></code></td>
</tr>
<tr class="row-odd"><td><code class="docutils literal notranslate"><span class="pre">{{</span> <span class="pre">ti</span> <span class="pre">}}</span></code></td>
<td>same as <code class="docutils literal notranslate"><span class="pre">{{</span> <span class="pre">task_instance</span> <span class="pre">}}</span></code></td>
</tr>
<tr class="row-even"><td><code class="docutils literal notranslate"><span class="pre">{{</span> <span class="pre">params</span> <span class="pre">}}</span></code></td>
<td>a reference to the user-defined params dictionary which can be overridden by
the dictionary passed through <code class="docutils literal notranslate"><span class="pre">trigger_dag</span> <span class="pre">-c</span></code> if you enabled
<code class="docutils literal notranslate"><span class="pre">dag_run_conf_overrides_params`</span> <span class="pre">in</span> <span class="pre">``airflow.cfg</span></code></td>
</tr>
<tr class="row-odd"><td><code class="docutils literal notranslate"><span class="pre">{{</span> <span class="pre">var.value.my_var</span> <span class="pre">}}</span></code></td>
<td>global defined variables represented as a dictionary</td>
</tr>
<tr class="row-even"><td><code class="docutils literal notranslate"><span class="pre">{{</span> <span class="pre">var.json.my_var.path</span> <span class="pre">}}</span></code></td>
<td>global defined variables represented as a dictionary
with deserialized JSON object, append the path to the
key within the JSON object</td>
</tr>
<tr class="row-odd"><td><code class="docutils literal notranslate"><span class="pre">{{</span> <span class="pre">task_instance_key_str</span> <span class="pre">}}</span></code></td>
<td>a unique, human-readable key to the task instance
formatted <code class="docutils literal notranslate"><span class="pre">{dag_id}_{task_id}_{ds}</span></code></td>
</tr>
<tr class="row-even"><td><code class="docutils literal notranslate"><span class="pre">{{</span> <span class="pre">conf</span> <span class="pre">}}</span></code></td>
<td>the full configuration object located at
<code class="docutils literal notranslate"><span class="pre">airflow.configuration.conf</span></code> which
represents the content of your
<code class="docutils literal notranslate"><span class="pre">airflow.cfg</span></code></td>
</tr>
<tr class="row-odd"><td><code class="docutils literal notranslate"><span class="pre">{{</span> <span class="pre">run_id</span> <span class="pre">}}</span></code></td>
<td>the <code class="docutils literal notranslate"><span class="pre">run_id</span></code> of the current DAG run</td>
</tr>
<tr class="row-even"><td><code class="docutils literal notranslate"><span class="pre">{{</span> <span class="pre">dag_run</span> <span class="pre">}}</span></code></td>
<td>a reference to the DagRun object</td>
</tr>
<tr class="row-odd"><td><code class="docutils literal notranslate"><span class="pre">{{</span> <span class="pre">test_mode</span> <span class="pre">}}</span></code></td>
<td>whether the task instance was called using
the CLI&#x2019;s test subcommand</td>
</tr>
</tbody>
</table>
<p>Note that you can access the object&#x2019;s attributes and methods with simple
dot notation. Here are some examples of what is possible:
<code class="docutils literal notranslate"><span class="pre">{{</span> <span class="pre">task.owner</span> <span class="pre">}}</span></code>, <code class="docutils literal notranslate"><span class="pre">{{</span> <span class="pre">task.task_id</span> <span class="pre">}}</span></code>, <code class="docutils literal notranslate"><span class="pre">{{</span> <span class="pre">ti.hostname</span> <span class="pre">}}</span></code>, &#x2026;
Refer to the models documentation for more information on the objects&#x2019;
attributes and methods.</p>
<p>The <code class="docutils literal notranslate"><span class="pre">var</span></code> template variable allows you to access variables defined in Airflow&#x2019;s
UI. You can access them as either plain-text or JSON. If you use JSON, you are
also able to walk nested structures, such as dictionaries like:
<code class="docutils literal notranslate"><span class="pre">{{</span> <span class="pre">var.json.my_dict_var.key1</span> <span class="pre">}}</span></code></p>
</div>
<div class="section" id="id13">
<h3 class="sigil_not_in_toc">Macros</h3>
<p>Macros are a way to expose objects to your templates and live under the
<code class="docutils literal notranslate"><span class="pre">macros</span></code> namespace in your templates.</p>
<p>A few commonly used libraries and methods are made available.</p>
<table border="1" class="docutils">
<colgroup>
<col width="44%">
<col width="56%">
</colgroup>
<thead valign="bottom">
<tr class="row-odd"><th class="head">Variable</th>
<th class="head">Description</th>
</tr>
</thead>
<tbody valign="top">
<tr class="row-even"><td><code class="docutils literal notranslate"><span class="pre">macros.datetime</span></code></td>
<td>The standard lib&#x2019;s <code class="docutils literal notranslate"><span class="pre">datetime.datetime</span></code></td>
</tr>
<tr class="row-odd"><td><code class="docutils literal notranslate"><span class="pre">macros.timedelta</span></code></td>
<td>The standard lib&#x2019;s <code class="docutils literal notranslate"><span class="pre">datetime.timedelta</span></code></td>
</tr>
<tr class="row-even"><td><code class="docutils literal notranslate"><span class="pre">macros.dateutil</span></code></td>
<td>A reference to the <code class="docutils literal notranslate"><span class="pre">dateutil</span></code> package</td>
</tr>
<tr class="row-odd"><td><code class="docutils literal notranslate"><span class="pre">macros.time</span></code></td>
<td>The standard lib&#x2019;s <code class="docutils literal notranslate"><span class="pre">time</span></code></td>
</tr>
<tr class="row-even"><td><code class="docutils literal notranslate"><span class="pre">macros.uuid</span></code></td>
<td>The standard lib&#x2019;s <code class="docutils literal notranslate"><span class="pre">uuid</span></code></td>
</tr>
<tr class="row-odd"><td><code class="docutils literal notranslate"><span class="pre">macros.random</span></code></td>
<td>The standard lib&#x2019;s <code class="docutils literal notranslate"><span class="pre">random</span></code></td>
</tr>
</tbody>
</table>
<p>Some airflow specific macros are also defined:</p>
<span class="target" id="module-airflow.macros"></span>
<pre>
airflow.macros.ds_add(ds, days)</pre>
<p>Add or subtract days from a YYYY-MM-DD</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>ds</strong> (<em>str</em>) &#x2013; anchor date in <code class="docutils literal notranslate"><span class="pre">YYYY-MM-DD</span></code> format to add to</li>
<li><strong>days</strong> (<em>int</em>) &#x2013; number of days to add to the ds, you can use negative values</li>
</ul>
</td>
</tr>
</tbody>
</table>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">ds_add</span><span class="p">(</span><span class="s1">&apos;2015-01-01&apos;</span><span class="p">,</span> <span class="mi">5</span><span class="p">)</span>
<span class="go">&apos;2015-01-06&apos;</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">ds_add</span><span class="p">(</span><span class="s1">&apos;2015-01-06&apos;</span><span class="p">,</span> <span class="o">-</span><span class="mi">5</span><span class="p">)</span>
<span class="go">&apos;2015-01-01&apos;</span>
</pre>
</div>
</div>




<pre>
airflow.macros.ds_format(ds, input_format, output_format)</pre>
<p>Takes an input string and outputs another string
as specified in the output format</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>ds</strong> (<em>str</em>) &#x2013; input string which contains a date</li>
<li><strong>input_format</strong> (<em>str</em>) &#x2013; input string format. E.g. %Y-%m-%d</li>
<li><strong>output_format</strong> (<em>str</em>) &#x2013; output string format  E.g. %Y-%m-%d</li>
</ul>
</td>
</tr>
</tbody>
</table>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">ds_format</span><span class="p">(</span><span class="s1">&apos;2015-01-01&apos;</span><span class="p">,</span> <span class="s2">&quot;%Y-%m-</span><span class="si">%d</span><span class="s2">&quot;</span><span class="p">,</span> <span class="s2">&quot;%m-</span><span class="si">%d</span><span class="s2">-%y&quot;</span><span class="p">)</span>
<span class="go">&apos;01-01-15&apos;</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">ds_format</span><span class="p">(</span><span class="s1">&apos;1/5/2015&apos;</span><span class="p">,</span> <span class="s2">&quot;%m/</span><span class="si">%d</span><span class="s2">/%Y&quot;</span><span class="p">,</span>  <span class="s2">&quot;%Y-%m-</span><span class="si">%d</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="go">&apos;2015-01-05&apos;</span>
</pre>
</div>
</div>




<pre>
airflow.macros.random() &#x2192; x in the interval [0, 1).</pre>




<pre>
airflow.macros.hive.closest_ds_partition(table, ds, before=True, schema=&apos;default&apos;, metastore_conn_id=&apos;metastore_default&apos;)</pre>
<p>This function finds the date in a list closest to the target date.
An optional parameter can be given to get the closest before or after.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first simple">
<li><strong>table</strong> (<em>str</em>) &#x2013; A hive table name</li>
<li><strong>ds</strong> (<em>datetime.date list</em>) &#x2013; A datestamp <code class="docutils literal notranslate"><span class="pre">%Y-%m-%d</span></code> e.g. <code class="docutils literal notranslate"><span class="pre">yyyy-mm-dd</span></code></li>
<li><strong>before</strong> (<em>bool</em><em> or </em><em>None</em>) &#x2013; closest before (True), after (False) or either side of ds</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th>
<td class="field-body"><p class="first">The closest date</p>
</td>
</tr>
<tr class="field-odd field"><th class="field-name">Return type:</th>
<td class="field-body"><p class="first last">str or None</p>
</td>
</tr>
</tbody>
</table>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">tbl</span> <span class="o">=</span> <span class="s1">&apos;airflow.static_babynames_partitioned&apos;</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">closest_ds_partition</span><span class="p">(</span><span class="n">tbl</span><span class="p">,</span> <span class="s1">&apos;2015-01-02&apos;</span><span class="p">)</span>
<span class="go">&apos;2015-01-01&apos;</span>
</pre>
</div>
</div>




<pre>
airflow.macros.hive.max_partition(table, schema=&apos;default&apos;, field=None, filter_map=None, metastore_conn_id=&apos;metastore_default&apos;)</pre>
<p>Gets the max partition for a table.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>schema</strong> (<em>string</em>) &#x2013; The hive schema the table lives in</li>
<li><strong>table</strong> (<em>string</em>) &#x2013; The hive table you are interested in, supports the dot
notation as in &#x201C;my_database.my_table&#x201D;, if a dot is found,
the schema param is disregarded</li>
<li><strong>metastore_conn_id</strong> (<em>string</em>) &#x2013; The hive connection you are interested in.
If your default is set you don&#x2019;t need to use this parameter.</li>
<li><strong>filter_map</strong> (<em>map</em>) &#x2013; partition_key:partition_value map used for partition filtering,
e.g. {&#x2018;key1&#x2019;: &#x2018;value1&#x2019;, &#x2018;key2&#x2019;: &#x2018;value2&#x2019;}.
Only partitions matching all partition_key:partition_value
pairs will be considered as candidates of max partition.</li>
<li><strong>field</strong> (<em>str</em>) &#x2013; the field to get the max value from. If there&#x2019;s only
one partition field, this will be inferred</li>
</ul>
</td>
</tr>
</tbody>
</table>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">max_partition</span><span class="p">(</span><span class="s1">&apos;airflow.static_babynames_partitioned&apos;</span><span class="p">)</span>
<span class="go">&apos;2015-01-01&apos;</span>
</pre>
</div>
</div>



</div>
</div>
<div class="section" id="models">
<span id="models-ref"></span><h2 class="sigil_not_in_toc">Models</h2>
<p>Models are built on top of the SQLAlchemy ORM Base class, and instances are
persisted in the database.</p>
<span class="target" id="module-airflow.models"></span>
<pre>
class airflow.models.BaseOperator(task_id, owner=&apos;Airflow&apos;, email=None, email_on_retry=True, email_on_failure=True, retries=0, retry_delay=datetime.timedelta(0, 300), retry_exponential_backoff=False, max_retry_delay=None, start_date=None, end_date=None, schedule_interval=None, depends_on_past=False, wait_for_downstream=False, dag=None, params=None, default_args=None, adhoc=False, priority_weight=1, weight_rule=u&apos;downstream&apos;, queue=&apos;default&apos;, pool=None, sla=None, execution_timeout=None, on_failure_callback=None, on_success_callback=None, on_retry_callback=None, trigger_rule=u&apos;all_success&apos;, resources=None, run_as_user=None, task_concurrency=None, executor_config=None, inlets=None, outlets=None, *args, **kwargs)</pre>
<p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.utils.log.logging_mixin.LoggingMixin</span></code></p>
<p>Abstract base class for all operators. Since operators create objects that
become nodes in the dag, BaseOperator contains many recursive methods for
dag crawling behavior. To derive this class, you are expected to override
the constructor as well as the &#x2018;execute&#x2019; method.</p>
<p>Operators derived from this class should perform or trigger certain tasks
synchronously (wait for completion). Example of operators could be an
operator that runs a Pig job (PigOperator), a sensor operator that
waits for a partition to land in Hive (HiveSensorOperator), or one that
moves data from Hive to MySQL (Hive2MySqlOperator). Instances of these
operators (tasks) target specific operations, running specific scripts,
functions or data transfers.</p>
<p>This class is abstract and shouldn&#x2019;t be instantiated. Instantiating a
class derived from this one results in the creation of a task object,
which ultimately becomes a node in DAG objects. Task dependencies should
be set by using the set_upstream and/or set_downstream methods.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>task_id</strong> (<em>string</em>) &#x2013; a unique, meaningful id for the task</li>
<li><strong>owner</strong> (<em>string</em>) &#x2013; the owner of the task, using the unix username is recommended</li>
<li><strong>retries</strong> (<em>int</em>) &#x2013; the number of retries that should be performed before
failing the task</li>
<li><strong>retry_delay</strong> (<em>timedelta</em>) &#x2013; delay between retries</li>
<li><strong>retry_exponential_backoff</strong> (<em>bool</em>) &#x2013; allow progressive longer waits between
retries by using exponential backoff algorithm on retry delay (delay
will be converted into seconds)</li>
<li><strong>max_retry_delay</strong> (<em>timedelta</em>) &#x2013; maximum delay interval between retries</li>
<li><strong>start_date</strong> (<em>datetime</em>) &#x2013; The <code class="docutils literal notranslate"><span class="pre">start_date</span></code> for the task, determines
the <code class="docutils literal notranslate"><span class="pre">execution_date</span></code> for the first task instance. The best practice
is to have the start_date rounded
to your DAG&#x2019;s <code class="docutils literal notranslate"><span class="pre">schedule_interval</span></code>. Daily jobs have their start_date
some day at 00:00:00, hourly jobs have their start_date at 00:00
of a specific hour. Note that Airflow simply looks at the latest
<code class="docutils literal notranslate"><span class="pre">execution_date</span></code> and adds the <code class="docutils literal notranslate"><span class="pre">schedule_interval</span></code> to determine
the next <code class="docutils literal notranslate"><span class="pre">execution_date</span></code>. It is also very important
to note that different tasks&#x2019; dependencies
need to line up in time. If task A depends on task B and their
start_date are offset in a way that their execution_date don&#x2019;t line
up, A&#x2019;s dependencies will never be met. If you are looking to delay
a task, for example running a daily task at 2AM, look into the
<code class="docutils literal notranslate"><span class="pre">TimeSensor</span></code> and <code class="docutils literal notranslate"><span class="pre">TimeDeltaSensor</span></code>. We advise against using
dynamic <code class="docutils literal notranslate"><span class="pre">start_date</span></code> and recommend using fixed ones. Read the
FAQ entry about start_date for more information.</li>
<li><strong>end_date</strong> (<em>datetime</em>) &#x2013; if specified, the scheduler won&#x2019;t go beyond this date</li>
<li><strong>depends_on_past</strong> (<em>bool</em>) &#x2013; when set to true, task instances will run
sequentially while relying on the previous task&#x2019;s schedule to
succeed. The task instance for the start_date is allowed to run.</li>
<li><strong>wait_for_downstream</strong> (<em>bool</em>) &#x2013; when set to true, an instance of task
X will wait for tasks immediately downstream of the previous instance
of task X to finish successfully before it runs. This is useful if the
different instances of a task X alter the same asset, and this asset
is used by tasks downstream of task X. Note that depends_on_past
is forced to True wherever wait_for_downstream is used.</li>
<li><strong>queue</strong> (<em>str</em>) &#x2013; which queue to target when running this job. Not
all executors implement queue management, the CeleryExecutor
does support targeting specific queues.</li>
<li><strong>dag</strong> (<a class="reference internal" href="#airflow.models.DAG" title="airflow.models.DAG"><em>DAG</em></a>) &#x2013; a reference to the dag the task is attached to (if any)</li>
<li><strong>priority_weight</strong> (<em>int</em>) &#x2013; priority weight of this task against other task.
This allows the executor to trigger higher priority tasks before
others when things get backed up.</li>
<li><strong>weight_rule</strong> (<em>str</em>) &#x2013; weighting method used for the effective total
priority weight of the task. Options are:
<code class="docutils literal notranslate"><span class="pre">{</span> <span class="pre">downstream</span> <span class="pre">|</span> <span class="pre">upstream</span> <span class="pre">|</span> <span class="pre">absolute</span> <span class="pre">}</span></code> default is <code class="docutils literal notranslate"><span class="pre">downstream</span></code>
When set to <code class="docutils literal notranslate"><span class="pre">downstream</span></code> the effective weight of the task is the
aggregate sum of all downstream descendants. As a result, upstream
tasks will have higher weight and will be scheduled more aggressively
when using positive weight values. This is useful when you have
multiple dag run instances and desire to have all upstream tasks to
complete for all runs before each dag can continue processing
downstream tasks. When set to <code class="docutils literal notranslate"><span class="pre">upstream</span></code> the effective weight is the
aggregate sum of all upstream ancestors. This is the opposite where
downtream tasks have higher weight and will be scheduled more
aggressively when using positive weight values. This is useful when you
have multiple dag run instances and prefer to have each dag complete
before starting upstream tasks of other dags.  When set to
<code class="docutils literal notranslate"><span class="pre">absolute</span></code>, the effective weight is the exact <code class="docutils literal notranslate"><span class="pre">priority_weight</span></code>
specified without additional weighting. You may want to do this when
you know exactly what priority weight each task should have.
Additionally, when set to <code class="docutils literal notranslate"><span class="pre">absolute</span></code>, there is bonus effect of
significantly speeding up the task creation process as for very large
DAGS. Options can be set as string or using the constants defined in
the static class <code class="docutils literal notranslate"><span class="pre">airflow.utils.WeightRule</span></code></li>
<li><strong>pool</strong> (<em>str</em>) &#x2013; the slot pool this task should run in, slot pools are a
way to limit concurrency for certain tasks</li>
<li><strong>sla</strong> (<em>datetime.timedelta</em>) &#x2013; time by which the job is expected to succeed. Note that
this represents the <code class="docutils literal notranslate"><span class="pre">timedelta</span></code> after the period is closed. For
example if you set an SLA of 1 hour, the scheduler would send an email
soon after 1:00AM on the <code class="docutils literal notranslate"><span class="pre">2016-01-02</span></code> if the <code class="docutils literal notranslate"><span class="pre">2016-01-01</span></code> instance
has not succeeded yet.
The scheduler pays special attention for jobs with an SLA and
sends alert
emails for sla misses. SLA misses are also recorded in the database
for future reference. All tasks that share the same SLA time
get bundled in a single email, sent soon after that time. SLA
notification are sent once and only once for each task instance.</li>
<li><strong>execution_timeout</strong> (<em>datetime.timedelta</em>) &#x2013; max time allowed for the execution of
this task instance, if it goes beyond it will raise and fail.</li>
<li><strong>on_failure_callback</strong> (<em>callable</em>) &#x2013; a function to be called when a task instance
of this task fails. a context dictionary is passed as a single
parameter to this function. Context contains references to related
objects to the task instance and is documented under the macros
section of the API.</li>
<li><strong>on_retry_callback</strong> &#x2013; much like the <code class="docutils literal notranslate"><span class="pre">on_failure_callback</span></code> except
that it is executed when retries occur.</li>
<li><strong>on_success_callback</strong> (<em>callable</em>) &#x2013; much like the <code class="docutils literal notranslate"><span class="pre">on_failure_callback</span></code> except
that it is executed when the task succeeds.</li>
<li><strong>trigger_rule</strong> (<em>str</em>) &#x2013; defines the rule by which dependencies are applied
for the task to get triggered. Options are:
<code class="docutils literal notranslate"><span class="pre">{</span> <span class="pre">all_success</span> <span class="pre">|</span> <span class="pre">all_failed</span> <span class="pre">|</span> <span class="pre">all_done</span> <span class="pre">|</span> <span class="pre">one_success</span> <span class="pre">|</span>
<span class="pre">one_failed</span> <span class="pre">|</span> <span class="pre">dummy}</span></code>
default is <code class="docutils literal notranslate"><span class="pre">all_success</span></code>. Options can be set as string or
using the constants defined in the static class
<code class="docutils literal notranslate"><span class="pre">airflow.utils.TriggerRule</span></code></li>
<li><strong>resources</strong> (<em>dict</em>) &#x2013; A map of resource parameter names (the argument names of the
Resources constructor) to their values.</li>
<li><strong>run_as_user</strong> (<em>str</em>) &#x2013; unix username to impersonate while running the task</li>
<li><strong>task_concurrency</strong> (<em>int</em>) &#x2013; When set, a task will be able to limit the concurrent
runs across execution_dates</li>
<li><strong>executor_config</strong> (<em>dict</em>) &#x2013; <p>Additional task-level configuration parameters that are
interpreted by a specific executor. Parameters are namespaced by the name of
executor.
<a href="#id14"><span class="problematic" id="id15">``</span></a>example: to run this task in a specific docker container through
the KubernetesExecutor
MyOperator(&#x2026;,</p>
<blockquote>
<div>executor_config={
&#x201C;KubernetesExecutor&#x201D;:<blockquote>
<div>{&#x201C;image&#x201D;: &#x201C;myCustomDockerImage&#x201D;}
}</div>
</blockquote>
</div>
</blockquote>
<p>)``</p>
</li>
</ul>
</td>
</tr>
</tbody>
</table>

<pre>
clear(**kwargs)</pre>
<p>Clears the state of task instances associated with the task, following
the parameters specified.</p>




<pre>
dag</pre>
<p>Returns the Operator&#x2019;s DAG if set, otherwise raises an error</p>




<pre>
deps</pre>
<p>Returns the list of dependencies for the operator. These differ from execution
context dependencies in that they are specific to tasks and can be
extended/overridden by subclasses.</p>




<pre>
downstream_list</pre>
<p>@property: list of tasks directly downstream</p>




<pre>
execute(context)</pre>
<p>This is the main method to derive when creating an operator.
Context is the same dictionary used as when rendering jinja templates.</p>
<p>Refer to get_template_context for more context.</p>




<pre>
get_direct_relative_ids(upstream=False)</pre>
<p>Get the direct relative ids to the current task, upstream or
downstream.</p>




<pre>
get_direct_relatives(upstream=False)</pre>
<p>Get the direct relatives to the current task, upstream or
downstream.</p>




<pre>
get_flat_relative_ids(upstream=False, found_descendants=None)</pre>
<p>Get a flat list of relatives&#x2019; ids, either upstream or downstream.</p>




<pre>
get_flat_relatives(upstream=False)</pre>
<p>Get a flat list of relatives, either upstream or downstream.</p>




<pre>
get_task_instances(session, start_date=None, end_date=None)</pre>
<p>Get a set of task instance related to this task for a specific date
range.</p>




<pre>
has_dag()</pre>
<p>Returns True if the Operator has been assigned to a DAG.</p>




<pre>
on_kill()</pre>
<p>Override this method to cleanup subprocesses when a task instance
gets killed. Any use of the threading, subprocess or multiprocessing
module within an operator needs to be cleaned up or it will leave
ghost processes behind.</p>




<pre>
post_execute(context, *args, **kwargs)</pre>
<p>This hook is triggered right after self.execute() is called.
It is passed the execution context and any results returned by the
operator.</p>




<pre>
pre_execute(context, *args, **kwargs)</pre>
<p>This hook is triggered right before self.execute() is called.</p>




<pre>
prepare_template()</pre>
<p>Hook that is triggered after the templated fields get replaced
by their content. If you need your operator to alter the
content of the file before the template is rendered,
it should override this method to do so.</p>




<pre>
render_template(attr, content, context)</pre>
<p>Renders a template either from a file or directly in a field, and returns
the rendered result.</p>




<pre>
render_template_from_field(attr, content, context, jinja_env)</pre>
<p>Renders a template from a field. If the field is a string, it will
simply render the string and return the result. If it is a collection or
nested set of collections, it will traverse the structure and render
all strings in it.</p>




<pre>
run(start_date=None, end_date=None, ignore_first_depends_on_past=False, ignore_ti_state=False, mark_success=False)</pre>
<p>Run a set of task instances for a date range.</p>




<pre>
schedule_interval</pre>
<p>The schedule interval of the DAG always wins over individual tasks so
that tasks within a DAG always line up. The task still needs a
schedule_interval as it may not be attached to a DAG.</p>




<pre>
set_downstream(task_or_task_list)</pre>
<p>Set a task or a task list to be directly downstream from the current
task.</p>




<pre>
set_upstream(task_or_task_list)</pre>
<p>Set a task or a task list to be directly upstream from the current
task.</p>




<pre>
upstream_list</pre>
<p>@property: list of tasks directly upstream</p>




<pre>
xcom_pull(context, task_ids=None, dag_id=None, key=u&apos;return_value&apos;, include_prior_dates=None)</pre>
<p>See TaskInstance.xcom_pull()</p>




<pre>
xcom_push(context, key, value, execution_date=None)</pre>
<p>See TaskInstance.xcom_push()</p>







<pre>
class airflow.models.Chart(**kwargs)</pre>
<p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">sqlalchemy.ext.declarative.api.Base</span></code></p>




<pre>
class airflow.models.Connection(conn_id=None, conn_type=None, host=None, login=None, password=None, schema=None, port=None, extra=None, uri=None)</pre>
<p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">sqlalchemy.ext.declarative.api.Base</span></code>, <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.utils.log.logging_mixin.LoggingMixin</span></code></p>
<p>Placeholder to store information about different database instances
connection information. The idea here is that scripts use references to
database instances (conn_id) instead of hard coding hostname, logins and
passwords when using operators or hooks.</p>

<pre>
extra_dejson</pre>
<p>Returns the extra property by deserializing json.</p>







<pre>
class airflow.models.DAG(dag_id, description=u&apos;&apos;, schedule_interval=datetime.timedelta(1), start_date=None, end_date=None, full_filepath=None, template_searchpath=None, user_defined_macros=None, user_defined_filters=None, default_args=None, concurrency=16, max_active_runs=16, dagrun_timeout=None, sla_miss_callback=None, default_view=u&apos;tree&apos;, orientation=&apos;LR&apos;, catchup=True, on_success_callback=None, on_failure_callback=None, params=None)</pre>
<p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.dag.base_dag.BaseDag</span></code>, <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.utils.log.logging_mixin.LoggingMixin</span></code></p>
<p>A dag (directed acyclic graph) is a collection of tasks with directional
dependencies. A dag also has a schedule, a start end an end date
(optional). For each schedule, (say daily or hourly), the DAG needs to run
each individual tasks as their dependencies are met. Certain tasks have
the property of depending on their own past, meaning that they can&#x2019;t run
until their previous schedule (and upstream tasks) are completed.</p>
<p>DAGs essentially act as namespaces for tasks. A task_id can only be
added once to a DAG.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>dag_id</strong> (<em>string</em>) &#x2013; The id of the DAG</li>
<li><strong>description</strong> (<em>string</em>) &#x2013; The description for the DAG to e.g. be shown on the webserver</li>
<li><strong>schedule_interval</strong> (<em>datetime.timedelta</em><em> or
</em><em>dateutil.relativedelta.relativedelta</em><em> or </em><em>str that acts as a cron
expression</em>) &#x2013; Defines how often that DAG runs, this
timedelta object gets added to your latest task instance&#x2019;s
execution_date to figure out the next schedule</li>
<li><strong>start_date</strong> (<em>datetime.datetime</em>) &#x2013; The timestamp from which the scheduler will
attempt to backfill</li>
<li><strong>end_date</strong> (<em>datetime.datetime</em>) &#x2013; A date beyond which your DAG won&#x2019;t run, leave to None
for open ended scheduling</li>
<li><strong>template_searchpath</strong> (<em>string</em><em> or </em><em>list of stings</em>) &#x2013; This list of folders (non relative)
defines where jinja will look for your templates. Order matters.
Note that jinja/airflow includes the path of your DAG file by
default</li>
<li><strong>user_defined_macros</strong> (<em>dict</em>) &#x2013; a dictionary of macros that will be exposed
in your jinja templates. For example, passing <code class="docutils literal notranslate"><span class="pre">dict(foo=&apos;bar&apos;)</span></code>
to this argument allows you to <code class="docutils literal notranslate"><span class="pre">{{</span> <span class="pre">foo</span> <span class="pre">}}</span></code> in all jinja
templates related to this DAG. Note that you can pass any
type of object here.</li>
<li><strong>user_defined_filters</strong> (<em>dict</em>) &#x2013; a dictionary of filters that will be exposed
in your jinja templates. For example, passing
<code class="docutils literal notranslate"><span class="pre">dict(hello=lambda</span> <span class="pre">name:</span> <span class="pre">&apos;Hello</span> <span class="pre">%s&apos;</span> <span class="pre">%</span> <span class="pre">name)</span></code> to this argument allows
you to <code class="docutils literal notranslate"><span class="pre">{{</span> <span class="pre">&apos;world&apos;</span> <span class="pre">|</span> <span class="pre">hello</span> <span class="pre">}}</span></code> in all jinja templates related to
this DAG.</li>
<li><strong>default_args</strong> (<em>dict</em>) &#x2013; A dictionary of default parameters to be used
as constructor keyword parameters when initialising operators.
Note that operators have the same hook, and precede those defined
here, meaning that if your dict contains <cite>&#x2018;depends_on_past&#x2019;: True</cite>
here and <cite>&#x2018;depends_on_past&#x2019;: False</cite> in the operator&#x2019;s call
<cite>default_args</cite>, the actual value will be <cite>False</cite>.</li>
<li><strong>params</strong> (<em>dict</em>) &#x2013; a dictionary of DAG level parameters that are made
accessible in templates, namespaced under <cite>params</cite>. These
params can be overridden at the task level.</li>
<li><strong>concurrency</strong> (<em>int</em>) &#x2013; the number of task instances allowed to run
concurrently</li>
<li><strong>max_active_runs</strong> (<em>int</em>) &#x2013; maximum number of active DAG runs, beyond this
number of DAG runs in a running state, the scheduler won&#x2019;t create
new active DAG runs</li>
<li><strong>dagrun_timeout</strong> (<em>datetime.timedelta</em>) &#x2013; specify how long a DagRun should be up before
timing out / failing, so that new DagRuns can be created</li>
<li><strong>sla_miss_callback</strong> (<em>types.FunctionType</em>) &#x2013; specify a function to call when reporting SLA
timeouts.</li>
<li><strong>default_view</strong> (<em>string</em>) &#x2013; Specify DAG default view (tree, graph, duration,
gantt, landing_times)</li>
<li><strong>orientation</strong> (<em>string</em>) &#x2013; Specify DAG orientation in graph view (LR, TB, RL, BT)</li>
<li><strong>catchup</strong> (<em>bool</em>) &#x2013; Perform scheduler catchup (or only run latest)? Defaults to True</li>
<li><strong>on_failure_callback</strong> (<em>callable</em>) &#x2013; A function to be called when a DagRun of this dag fails.
A context dictionary is passed as a single parameter to this function.</li>
<li><strong>on_success_callback</strong> (<em>callable</em>) &#x2013; Much like the <code class="docutils literal notranslate"><span class="pre">on_failure_callback</span></code> except
that it is executed when the dag succeeds.</li>
</ul>
</td>
</tr>
</tbody>
</table>

<pre>
add_task(task)</pre>
<p>Add a task to the DAG</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><strong>task</strong> (<em>task</em>) &#x2013; the task you want to add</td>
</tr>
</tbody>
</table>




<pre>
add_tasks(tasks)</pre>
<p>Add a list of tasks to the DAG</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><strong>tasks</strong> (<em>list of tasks</em>) &#x2013; a lit of tasks you want to add</td>
</tr>
</tbody>
</table>




<pre>
clear(**kwargs)</pre>
<p>Clears a set of task instances associated with the current dag for
a specified date range.</p>




<pre>
cli()</pre>
<p>Exposes a CLI specific to this DAG</p>




<pre>
concurrency_reached</pre>
<p>Returns a boolean indicating whether the concurrency limit for this DAG
has been reached</p>




<pre>
create_dagrun(**kwargs)</pre>
<p>Creates a dag run from this dag including the tasks associated with this dag.
Returns the dag run.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>run_id</strong> (<em>string</em>) &#x2013; defines the the run id for this dag run</li>
<li><strong>execution_date</strong> (<em>datetime</em>) &#x2013; the execution date of this dag run</li>
<li><strong>state</strong> (<em>State</em>) &#x2013; the state of the dag run</li>
<li><strong>start_date</strong> (<em>datetime</em>) &#x2013; the date this dag run should be evaluated</li>
<li><strong>external_trigger</strong> (<em>bool</em>) &#x2013; whether this dag run is externally triggered</li>
<li><strong>session</strong> (<em>Session</em>) &#x2013; database session</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
static deactivate_stale_dags(*args, **kwargs)</pre>
<p>Deactivate any DAGs that were last touched by the scheduler before
the expiration date. These DAGs were likely deleted.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><strong>expiration_date</strong> (<em>datetime</em>) &#x2013; set inactive DAGs that were touched before this
time</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th>
<td class="field-body">None</td>
</tr>
</tbody>
</table>




<pre>
static deactivate_unknown_dags(*args, **kwargs)</pre>
<p>Given a list of known DAGs, deactivate any other DAGs that are
marked as active in the ORM</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><strong>active_dag_ids</strong> (<em>list</em><em>[</em><em>unicode</em><em>]</em>) &#x2013; list of DAG IDs that are active</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th>
<td class="field-body">None</td>
</tr>
</tbody>
</table>




<pre>
filepath</pre>
<p>File location of where the dag object is instantiated</p>




<pre>
folder</pre>
<p>Folder location of where the dag object is instantiated</p>




<pre>
following_schedule(dttm)</pre>
<p>Calculates the following schedule for this dag in local time</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><strong>dttm</strong> &#x2013; utc datetime</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th>
<td class="field-body">utc datetime</td>
</tr>
</tbody>
</table>




<pre>
get_active_runs(**kwargs)</pre>
<p>Returns a list of dag run execution dates currently running</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><strong>session</strong> &#x2013; </td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th>
<td class="field-body">List of execution dates</td>
</tr>
</tbody>
</table>




<pre>
get_dagrun(**kwargs)</pre>
<p>Returns the dag run for a given execution date if it exists, otherwise
none.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first simple">
<li><strong>execution_date</strong> &#x2013; The execution date of the DagRun to find.</li>
<li><strong>session</strong> &#x2013; </li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th>
<td class="field-body"><p class="first last">The DagRun if found, otherwise None.</p>
</td>
</tr>
</tbody>
</table>




<pre>
get_last_dagrun(**kwargs)</pre>
<p>Returns the last dag run for this dag, None if there was none.
Last dag run can be any type of run eg. scheduled or backfilled.
Overridden DagRuns are ignored</p>




<pre>
get_num_active_runs(**kwargs)</pre>
<p>Returns the number of active &#x201C;running&#x201D; dag runs</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first simple">
<li><strong>external_trigger</strong> (<em>bool</em>) &#x2013; True for externally triggered active dag runs</li>
<li><strong>session</strong> &#x2013; </li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th>
<td class="field-body"><p class="first last">number greater than 0 for active dag runs</p>
</td>
</tr>
</tbody>
</table>




<pre>
static get_num_task_instances(*args, **kwargs)</pre>
<p>Returns the number of task instances in the given DAG.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first simple">
<li><strong>session</strong> &#x2013; ORM session</li>
<li><strong>dag_id</strong> (<em>unicode</em>) &#x2013; ID of the DAG to get the task concurrency of</li>
<li><strong>task_ids</strong> (<em>list</em><em>[</em><em>unicode</em><em>]</em>) &#x2013; A list of valid task IDs for the given DAG</li>
<li><strong>states</strong> (<em>list</em><em>[</em><em>state</em><em>]</em>) &#x2013; A list of states to filter by if supplied</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th>
<td class="field-body"><p class="first">The number of running tasks</p>
</td>
</tr>
<tr class="field-odd field"><th class="field-name">Return type:</th>
<td class="field-body"><p class="first last">int</p>
</td>
</tr>
</tbody>
</table>




<pre>
get_run_dates(start_date, end_date=None)</pre>
<p>Returns a list of dates between the interval received as parameter using this
dag&#x2019;s schedule interval. Returned dates can be used for execution dates.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first simple">
<li><strong>start_date</strong> (<em>datetime</em>) &#x2013; the start date of the interval</li>
<li><strong>end_date</strong> (<em>datetime</em>) &#x2013; the end date of the interval, defaults to timezone.utcnow()</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th>
<td class="field-body"><p class="first">a list of dates within the interval following the dag&#x2019;s schedule</p>
</td>
</tr>
<tr class="field-odd field"><th class="field-name">Return type:</th>
<td class="field-body"><p class="first last">list</p>
</td>
</tr>
</tbody>
</table>




<pre>
get_template_env()</pre>
<p>Returns a jinja2 Environment while taking into account the DAGs
template_searchpath, user_defined_macros and user_defined_filters</p>




<pre>
handle_callback(**kwargs)</pre>
<p>Triggers the appropriate callback depending on the value of success, namely the
on_failure_callback or on_success_callback. This method gets the context of a
single TaskInstance part of this DagRun and passes that to the callable along
with a &#x2018;reason&#x2019;, primarily to differentiate DagRun failures.
.. note:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>The logs end up in $AIRFLOW_HOME/logs/scheduler/latest/PROJECT/DAG_FILE.py.log
</pre>
</div>
</div>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>dagrun</strong> &#x2013; DagRun object</li>
<li><strong>success</strong> &#x2013; Flag to specify if failure or success callback should be called</li>
<li><strong>reason</strong> &#x2013; Completion reason</li>
<li><strong>session</strong> &#x2013; Database session</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
is_paused</pre>
<p>Returns a boolean indicating whether this DAG is paused</p>




<pre>
latest_execution_date</pre>
<p>Returns the latest date for which at least one dag run exists</p>




<pre>
normalize_schedule(dttm)</pre>
<p>Returns dttm + interval unless dttm is first interval then it returns dttm</p>




<pre>
previous_schedule(dttm)</pre>
<p>Calculates the previous schedule for this dag in local time</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><strong>dttm</strong> &#x2013; utc datetime</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th>
<td class="field-body">utc datetime</td>
</tr>
</tbody>
</table>




<pre>
run(start_date=None, end_date=None, mark_success=False, local=False, executor=None, donot_pickle=False, ignore_task_deps=False, ignore_first_depends_on_past=False, pool=None, delay_on_limit_secs=1.0, verbose=False, conf=None, rerun_failed_tasks=False)</pre>
<p>Runs the DAG.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>start_date</strong> (<em>datetime</em>) &#x2013; the start date of the range to run</li>
<li><strong>end_date</strong> (<em>datetime</em>) &#x2013; the end date of the range to run</li>
<li><strong>mark_success</strong> (<em>bool</em>) &#x2013; True to mark jobs as succeeded without running them</li>
<li><strong>local</strong> (<em>bool</em>) &#x2013; True to run the tasks using the LocalExecutor</li>
<li><strong>executor</strong> (<em>BaseExecutor</em>) &#x2013; The executor instance to run the tasks</li>
<li><strong>donot_pickle</strong> (<em>bool</em>) &#x2013; True to avoid pickling DAG object and send to workers</li>
<li><strong>ignore_task_deps</strong> (<em>bool</em>) &#x2013; True to skip upstream tasks</li>
<li><strong>ignore_first_depends_on_past</strong> (<em>bool</em>) &#x2013; True to ignore depends_on_past
dependencies for the first set of tasks only</li>
<li><strong>pool</strong> (<em>string</em>) &#x2013; Resource pool to use</li>
<li><strong>delay_on_limit_secs</strong> (<em>float</em>) &#x2013; Time in seconds to wait before next attempt to run
dag run when max_active_runs limit has been reached</li>
<li><strong>verbose</strong> (<em>boolean</em>) &#x2013; Make logging output more verbose</li>
<li><strong>conf</strong> (<em>dict</em>) &#x2013; user defined dictionary passed from CLI</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
set_dependency(upstream_task_id, downstream_task_id)</pre>
<p>Simple utility method to set dependency between two tasks that
already have been added to the DAG using add_task()</p>




<pre>
sub_dag(task_regex, include_downstream=False, include_upstream=True)</pre>
<p>Returns a subset of the current dag as a deep copy of the current dag
based on a regex that should match one or many tasks, and includes
upstream and downstream neighbours based on the flag passed.</p>




<pre>
subdags</pre>
<p>Returns a list of the subdag objects associated to this DAG</p>




<pre>
sync_to_db(**kwargs)</pre>
<p>Save attributes about this DAG to the DB. Note that this method
can be called for both DAGs and SubDAGs. A SubDag is actually a
SubDagOperator.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first simple">
<li><strong>dag</strong> (<a class="reference internal" href="#airflow.models.DAG" title="airflow.models.DAG"><em>DAG</em></a>) &#x2013; the DAG object to save to the DB</li>
<li><strong>sync_time</strong> (<em>datetime</em>) &#x2013; The time that the DAG should be marked as sync&#x2019;ed</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th>
<td class="field-body"><p class="first last">None</p>
</td>
</tr>
</tbody>
</table>




<pre>
test_cycle()</pre>
<p>Check to see if there are any cycles in the DAG. Returns False if no cycle found,
otherwise raises exception.</p>




<pre>
topological_sort()</pre>
<p>Sorts tasks in topographical order, such that a task comes after any of its
upstream dependencies.</p>
<p>Heavily inspired by:
<a class="reference external" href="http://blog.jupo.org/2012/04/06/topological-sorting-acyclic-directed-graphs/">http://blog.jupo.org/2012/04/06/topological-sorting-acyclic-directed-graphs/</a></p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Returns:</th>
<td class="field-body">list of tasks in topological order</td>
</tr>
</tbody>
</table>




<pre>
tree_view()</pre>
<p>Shows an ascii tree representation of the DAG</p>







<pre>
class airflow.models.DagBag(dag_folder=None, executor=None, include_examples=False)</pre>
<p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.dag.base_dag.BaseDagBag</span></code>, <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.utils.log.logging_mixin.LoggingMixin</span></code></p>
<p>A dagbag is a collection of dags, parsed out of a folder tree and has high
level configuration settings, like what database to use as a backend and
what executor to use to fire off tasks. This makes it easier to run
distinct environments for say production and development, tests, or for
different teams or security profiles. What would have been system level
settings are now dagbag level so that one system can run multiple,
independent settings sets.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>dag_folder</strong> (<em>unicode</em>) &#x2013; the folder to scan to find DAGs</li>
<li><strong>executor</strong> &#x2013; the executor to use when executing task instances
in this DagBag</li>
<li><strong>include_examples</strong> (<em>bool</em>) &#x2013; whether to include the examples that ship
with airflow or not</li>
<li><strong>has_logged</strong> &#x2013; an instance boolean that gets flipped from False to True after a
file has been skipped. This is to prevent overloading the user with logging
messages about skipped files. Therefore only once per DagBag is a file logged
being skipped.</li>
</ul>
</td>
</tr>
</tbody>
</table>

<pre>
bag_dag(dag, parent_dag, root_dag)</pre>
<p>Adds the DAG into the bag, recurses into sub dags.
Throws AirflowDagCycleException if a cycle is detected in this dag or its subdags</p>




<pre>
collect_dags(dag_folder=None, only_if_updated=True)</pre>
<p>Given a file path or a folder, this method looks for python modules,
imports them and adds them to the dagbag collection.</p>
<p>Note that if a .airflowignore file is found while processing,
the directory, it will behaves much like a .gitignore does,
ignoring files that match any of the regex patterns specified
in the file. <strong>Note</strong>: The patterns in .airflowignore are treated as
un-anchored regexes, not shell-like glob patterns.</p>




<pre>
dagbag_report()</pre>
<p>Prints a report around DagBag loading stats</p>




<pre>
get_dag(dag_id)</pre>
<p>Gets the DAG out of the dictionary, and refreshes it if expired</p>




<pre>
kill_zombies(**kwargs)</pre>
<p>Fails tasks that haven&#x2019;t had a heartbeat in too long</p>




<pre>
process_file(filepath, only_if_updated=True, safe_mode=True)</pre>
<p>Given a path to a python module or zip file, this method imports
the module and look for dag objects within it.</p>




<pre>
size()</pre>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Returns:</th>
<td class="field-body">the amount of dags contained in this dagbag</td>
</tr>
</tbody>
</table>







<pre>
class airflow.models.DagModel(**kwargs)</pre>
<p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">sqlalchemy.ext.declarative.api.Base</span></code></p>




<pre>
class airflow.models.DagPickle(dag)</pre>
<p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">sqlalchemy.ext.declarative.api.Base</span></code></p>
<p>Dags can originate from different places (user repos, master repo, &#x2026;)
and also get executed in different places (different executors). This
object represents a version of a DAG and becomes a source of truth for
a BackfillJob execution. A pickle is a native python serialized object,
and in this case gets stored in the database for the duration of the job.</p>
<p>The executors pick up the DagPickle id and read the dag definition from
the database.</p>




<pre>
class airflow.models.DagRun(**kwargs)</pre>
<p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">sqlalchemy.ext.declarative.api.Base</span></code>, <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.utils.log.logging_mixin.LoggingMixin</span></code></p>
<p>DagRun describes an instance of a Dag. It can be created
by the scheduler (for regular runs) or by an external trigger</p>

<pre>
static find(*args, **kwargs)</pre>
<p>Returns a set of dag runs for the given search criteria.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>dag_id</strong> (<em>integer</em><em>, </em><em>list</em>) &#x2013; the dag_id to find dag runs for</li>
<li><strong>run_id</strong> (<em>string</em>) &#x2013; defines the the run id for this dag run</li>
<li><strong>execution_date</strong> (<em>datetime</em>) &#x2013; the execution date</li>
<li><strong>state</strong> (<em>State</em>) &#x2013; the state of the dag run</li>
<li><strong>external_trigger</strong> (<em>bool</em>) &#x2013; whether this dag run is externally triggered</li>
<li><strong>no_backfills</strong> &#x2013; return no backfills (True), return all (False).</li>
</ul>
</td>
</tr>
</tbody>
</table>
<p>Defaults to False
:type no_backfills: bool
:param session: database session
:type session: Session</p>




<pre>
get_dag()</pre>
<p>Returns the Dag associated with this DagRun.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Returns:</th>
<td class="field-body">DAG</td>
</tr>
</tbody>
</table>




<pre>
classmethod get_latest_runs(**kwargs)</pre>
<p>Returns the latest DagRun for each DAG.</p>




<pre>
get_previous_dagrun(**kwargs)</pre>
<p>The previous DagRun, if there is one</p>




<pre>
get_previous_scheduled_dagrun(**kwargs)</pre>
<p>The previous, SCHEDULED DagRun, if there is one</p>




<pre>
static get_run(session, dag_id, execution_date)</pre>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first simple">
<li><strong>dag_id</strong> (<em>unicode</em>) &#x2013; DAG ID</li>
<li><strong>execution_date</strong> (<em>datetime</em>) &#x2013; execution date</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th>
<td class="field-body"><p class="first last">DagRun corresponding to the given dag_id and execution date</p>
</td>
</tr>
</tbody>
</table>
<p>if one exists. None otherwise.
:rtype: DagRun</p>




<pre>
get_task_instance(**kwargs)</pre>
<p>Returns the task instance specified by task_id for this dag run</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><strong>task_id</strong> &#x2013; the task id</td>
</tr>
</tbody>
</table>




<pre>
get_task_instances(**kwargs)</pre>
<p>Returns the task instances for this dag run</p>




<pre>
refresh_from_db(**kwargs)</pre>
<p>Reloads the current dagrun from the database
:param session: database session</p>




<pre>
update_state(**kwargs)</pre>
<p>Determines the overall state of the DagRun based on the state
of its TaskInstances.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Returns:</th>
<td class="field-body">State</td>
</tr>
</tbody>
</table>




<pre>
verify_integrity(**kwargs)</pre>
<p>Verifies the DagRun by checking for removed tasks or tasks that are not in the
database yet. It will set state to removed or add the task if required.</p>







<pre>
class airflow.models.DagStat(dag_id, state, count=0, dirty=False)</pre>
<p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">sqlalchemy.ext.declarative.api.Base</span></code></p>

<pre>
static create(*args, **kwargs)</pre>
<p>Creates the missing states the stats table for the dag specified</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first simple">
<li><strong>dag_id</strong> &#x2013; dag id of the dag to create stats for</li>
<li><strong>session</strong> &#x2013; database session</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th>
<td class="field-body"><p class="first last"></p>
</td>
</tr>
</tbody>
</table>




<pre>
static set_dirty(*args, **kwargs)</pre>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first simple">
<li><strong>dag_id</strong> &#x2013; the dag_id to mark dirty</li>
<li><strong>session</strong> &#x2013; database session</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th>
<td class="field-body"><p class="first last"></p>
</td>
</tr>
</tbody>
</table>




<pre>
static update(*args, **kwargs)</pre>
<p>Updates the stats for dirty/out-of-sync dags</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>dag_ids</strong> (<em>list</em>) &#x2013; dag_ids to be updated</li>
<li><strong>dirty_only</strong> (<em>bool</em>) &#x2013; only updated for marked dirty, defaults to True</li>
<li><strong>session</strong> (<em>Session</em>) &#x2013; db session to use</li>
</ul>
</td>
</tr>
</tbody>
</table>







<pre>
class airflow.models.ImportError(**kwargs)</pre>
<p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">sqlalchemy.ext.declarative.api.Base</span></code></p>




<pre>
exception airflow.models.InvalidFernetToken</pre>
<p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">exceptions.Exception</span></code></p>




<pre>
class airflow.models.KnownEvent(**kwargs)</pre>
<p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">sqlalchemy.ext.declarative.api.Base</span></code></p>




<pre>
class airflow.models.KnownEventType(**kwargs)</pre>
<p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">sqlalchemy.ext.declarative.api.Base</span></code></p>




<pre>
class airflow.models.KubeResourceVersion(**kwargs)</pre>
<p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">sqlalchemy.ext.declarative.api.Base</span></code></p>




<pre>
class airflow.models.KubeWorkerIdentifier(**kwargs)</pre>
<p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">sqlalchemy.ext.declarative.api.Base</span></code></p>




<pre>
class airflow.models.Log(event, task_instance, owner=None, extra=None, **kwargs)</pre>
<p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">sqlalchemy.ext.declarative.api.Base</span></code></p>
<p>Used to actively log events to the database</p>




<pre>
class airflow.models.NullFernet</pre>
<p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">future.types.newobject.newobject</span></code></p>
<p>A &#x201C;Null&#x201D; encryptor class that doesn&#x2019;t encrypt or decrypt but that presents
a similar interface to Fernet.</p>
<p>The purpose of this is to make the rest of the code not have to know the
difference, and to only display the message once, not 20 times when
<cite>airflow initdb</cite> is ran.</p>




<pre>
class airflow.models.Pool(**kwargs)</pre>
<p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">sqlalchemy.ext.declarative.api.Base</span></code></p>

<pre>
open_slots(**kwargs)</pre>
<p>Returns the number of slots open at the moment</p>




<pre>
queued_slots(**kwargs)</pre>
<p>Returns the number of slots used at the moment</p>




<pre>
used_slots(**kwargs)</pre>
<p>Returns the number of slots used at the moment</p>







<pre>
class airflow.models.SlaMiss(**kwargs)</pre>
<p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">sqlalchemy.ext.declarative.api.Base</span></code></p>
<p>Model that stores a history of the SLA that have been missed.
It is used to keep track of SLA failures over time and to avoid double
triggering alert emails.</p>




<pre>
class airflow.models.TaskFail(task, execution_date, start_date, end_date)</pre>
<p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">sqlalchemy.ext.declarative.api.Base</span></code></p>
<p>TaskFail tracks the failed run durations of each task instance.</p>




<pre>
class airflow.models.TaskInstance(task, execution_date, state=None)</pre>
<p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">sqlalchemy.ext.declarative.api.Base</span></code>, <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.utils.log.logging_mixin.LoggingMixin</span></code></p>
<p>Task instances store the state of a task instance. This table is the
authority and single source of truth around what tasks have run and the
state they are in.</p>
<p>The SqlAlchemy model doesn&#x2019;t have a SqlAlchemy foreign key to the task or
dag model deliberately to have more control over transactions.</p>
<p>Database transactions on this table should insure double triggers and
any confusion around what task instances are or aren&#x2019;t ready to run
even while multiple schedulers may be firing task instances.</p>

<pre>
are_dependencies_met(**kwargs)</pre>
<p>Returns whether or not all the conditions are met for this task instance to be run
given the context for the dependencies (e.g. a task instance being force run from
the UI will ignore some dependencies).</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>dep_context</strong> (<em>DepContext</em>) &#x2013; The execution context that determines the dependencies that
should be evaluated.</li>
<li><strong>session</strong> (<em>Session</em>) &#x2013; database session</li>
<li><strong>verbose</strong> (<em>boolean</em>) &#x2013; whether log details on failed dependencies on
info or debug log level</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
are_dependents_done(**kwargs)</pre>
<p>Checks whether the dependents of this task instance have all succeeded.
This is meant to be used by wait_for_downstream.</p>
<p>This is useful when you do not want to start processing the next
schedule of a task until the dependents are done. For instance,
if the task DROPs and recreates a table.</p>




<pre>
clear_xcom_data(**kwargs)</pre>
<p>Clears all XCom data from the database for the task instance</p>




<pre>
command(mark_success=False, ignore_all_deps=False, ignore_depends_on_past=False, ignore_task_deps=False, ignore_ti_state=False, local=False, pickle_id=None, raw=False, job_id=None, pool=None, cfg_path=None)</pre>
<p>Returns a command that can be executed anywhere where airflow is
installed. This command is part of the message sent to executors by
the orchestrator.</p>




<pre>
command_as_list(mark_success=False, ignore_all_deps=False, ignore_task_deps=False, ignore_depends_on_past=False, ignore_ti_state=False, local=False, pickle_id=None, raw=False, job_id=None, pool=None, cfg_path=None)</pre>
<p>Returns a command that can be executed anywhere where airflow is
installed. This command is part of the message sent to executors by
the orchestrator.</p>




<pre>
current_state(**kwargs)</pre>
<p>Get the very latest state from the database, if a session is passed,
we use and looking up the state becomes part of the session, otherwise
a new session is used.</p>




<pre>
error(**kwargs)</pre>
<p>Forces the task instance&#x2019;s state to FAILED in the database.</p>




<pre>
static generate_command(dag_id, task_id, execution_date, mark_success=False, ignore_all_deps=False, ignore_depends_on_past=False, ignore_task_deps=False, ignore_ti_state=False, local=False, pickle_id=None, file_path=None, raw=False, job_id=None, pool=None, cfg_path=None)</pre>
<p>Generates the shell command required to execute this task instance.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first simple">
<li><strong>dag_id</strong> (<em>unicode</em>) &#x2013; DAG ID</li>
<li><strong>task_id</strong> (<em>unicode</em>) &#x2013; Task ID</li>
<li><strong>execution_date</strong> (<em>datetime</em>) &#x2013; Execution date for the task</li>
<li><strong>mark_success</strong> (<em>bool</em>) &#x2013; Whether to mark the task as successful</li>
<li><strong>ignore_all_deps</strong> (<em>boolean</em>) &#x2013; Ignore all ignorable dependencies.
Overrides the other ignore_* parameters.</li>
<li><strong>ignore_depends_on_past</strong> (<em>boolean</em>) &#x2013; Ignore depends_on_past parameter of DAGs
(e.g. for Backfills)</li>
<li><strong>ignore_task_deps</strong> (<em>boolean</em>) &#x2013; Ignore task-specific dependencies such as depends_on_past
and trigger rule</li>
<li><strong>ignore_ti_state</strong> (<em>boolean</em>) &#x2013; Ignore the task instance&#x2019;s previous failure/success</li>
<li><strong>local</strong> (<em>bool</em>) &#x2013; Whether to run the task locally</li>
<li><strong>pickle_id</strong> (<em>unicode</em>) &#x2013; If the DAG was serialized to the DB, the ID
associated with the pickled DAG</li>
<li><strong>file_path</strong> &#x2013; path to the file containing the DAG definition</li>
<li><strong>raw</strong> &#x2013; raw mode (needs more details)</li>
<li><strong>job_id</strong> &#x2013; job ID (needs more details)</li>
<li><strong>pool</strong> (<em>unicode</em>) &#x2013; the Airflow pool that the task should run in</li>
<li><strong>cfg_path</strong> (<em>basestring</em>) &#x2013; the Path to the configuration file</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th>
<td class="field-body"><p class="first last">shell command that can be used to run the task instance</p>
</td>
</tr>
</tbody>
</table>




<pre>
get_dagrun(**kwargs)</pre>
<p>Returns the DagRun for this TaskInstance</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><strong>session</strong> &#x2013; </td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th>
<td class="field-body">DagRun</td>
</tr>
</tbody>
</table>




<pre>
init_on_load()</pre>
<p>Initialize the attributes that aren&#x2019;t stored in the DB.</p>




<pre>
init_run_context(raw=False)</pre>
<p>Sets the log context.</p>




<pre>
is_eligible_to_retry()</pre>
<p>Is task instance is eligible for retry</p>




<pre>
is_premature</pre>
<p>Returns whether a task is in UP_FOR_RETRY state and its retry interval
has elapsed.</p>




<pre>
key</pre>
<p>Returns a tuple that identifies the task instance uniquely</p>




<pre>
next_retry_datetime()</pre>
<p>Get datetime of the next retry if the task instance fails. For exponential
backoff, retry_delay is used as base and will be converted to seconds.</p>




<pre>
pool_full(**kwargs)</pre>
<p>Returns a boolean as to whether the slot pool has room for this
task to run</p>




<pre>
previous_ti</pre>
<p>The task instance for the task that ran before this task instance</p>




<pre>
ready_for_retry()</pre>
<p>Checks on whether the task instance is in the right state and timeframe
to be retried.</p>




<pre>
refresh_from_db(**kwargs)</pre>
<p>Refreshes the task instance from the database based on the primary key</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><strong>lock_for_update</strong> &#x2013; if True, indicates that the database should
lock the TaskInstance (issuing a FOR UPDATE clause) until the
session is committed.</td>
</tr>
</tbody>
</table>




<pre>
try_number</pre>
<p>Return the try number that this task number will be when it is acutally
run.</p>
<p>If the TI is currently running, this will match the column in the
databse, in all othercases this will be incremenetd</p>




<pre>
xcom_pull(task_ids=None, dag_id=None, key=u&apos;return_value&apos;, include_prior_dates=False)</pre>
<p>Pull XComs that optionally meet certain criteria.</p>
<p>The default value for <cite>key</cite> limits the search to XComs
that were returned by other tasks (as opposed to those that were pushed
manually). To remove this filter, pass key=None (or any desired value).</p>
<p>If a single task_id string is provided, the result is the value of the
most recent matching XCom from that task_id. If multiple task_ids are
provided, a tuple of matching values is returned. None is returned
whenever no matches are found.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>key</strong> (<em>string</em>) &#x2013; A key for the XCom. If provided, only XComs with matching
keys will be returned. The default key is &#x2018;return_value&#x2019;, also
available as a constant XCOM_RETURN_KEY. This key is automatically
given to XComs returned by tasks (as opposed to being pushed
manually). To remove the filter, pass key=None.</li>
<li><strong>task_ids</strong> (<em>string</em><em> or </em><em>iterable of strings</em><em> (</em><em>representing task_ids</em><em>)</em>) &#x2013; Only XComs from tasks with matching ids will be
pulled. Can pass None to remove the filter.</li>
<li><strong>dag_id</strong> (<em>string</em>) &#x2013; If provided, only pulls XComs from this DAG.
If None (default), the DAG of the calling task is used.</li>
<li><strong>include_prior_dates</strong> (<em>bool</em>) &#x2013; If False, only XComs from the current
execution_date are returned. If True, XComs from previous dates
are returned as well.</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
xcom_push(key, value, execution_date=None)</pre>
<p>Make an XCom available for tasks to pull.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>key</strong> (<em>string</em>) &#x2013; A key for the XCom</li>
<li><strong>value</strong> (<em>any pickleable object</em>) &#x2013; A value for the XCom. The value is pickled and stored
in the database.</li>
<li><strong>execution_date</strong> (<em>datetime</em>) &#x2013; if provided, the XCom will not be visible until
this date. This can be used, for example, to send a message to a
task on a future date without it being immediately visible.</li>
</ul>
</td>
</tr>
</tbody>
</table>







<pre>
class airflow.models.User(**kwargs)</pre>
<p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">sqlalchemy.ext.declarative.api.Base</span></code></p>




<pre>
class airflow.models.Variable(**kwargs)</pre>
<p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">sqlalchemy.ext.declarative.api.Base</span></code>, <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.utils.log.logging_mixin.LoggingMixin</span></code></p>

<pre>
classmethod setdefault(key, default, deserialize_json=False)</pre>
<p>Like a Python builtin dict object, setdefault returns the current value
for a key, and if it isn&#x2019;t there, stores the default value and returns it.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>key</strong> (<em>String</em>) &#x2013; Dict key for this Variable</li>
<li><strong>default</strong> &#x2013; Default value to set and return if the variable</li>
</ul>
</td>
</tr>
</tbody>
</table>
<p>isn&#x2019;t already in the DB
:type default: Mixed
:param deserialize_json: Store this as a JSON encoded value in the DB</p>
<blockquote>
<div>and un-encode it when retrieving a value</div>
</blockquote>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Returns:</th>
<td class="field-body">Mixed</td>
</tr>
</tbody>
</table>







<pre>
class airflow.models.XCom(**kwargs)</pre>
<p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">sqlalchemy.ext.declarative.api.Base</span></code>, <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.utils.log.logging_mixin.LoggingMixin</span></code></p>
<p>Base class for XCom objects.</p>

<pre>
classmethod get_many(**kwargs)</pre>
<p>Retrieve an XCom value, optionally meeting certain criteria
TODO: &#x201C;pickling&#x201D; has been deprecated and JSON is preferred.</p>
<blockquote>
<div>&#x201C;pickling&#x201D; will be removed in Airflow 2.0.</div>
</blockquote>




<pre>
classmethod get_one(**kwargs)</pre>
<p>Retrieve an XCom value, optionally meeting certain criteria.
TODO: &#x201C;pickling&#x201D; has been deprecated and JSON is preferred.</p>
<blockquote>
<div>&#x201C;pickling&#x201D; will be removed in Airflow 2.0.</div>
</blockquote>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Returns:</th>
<td class="field-body">XCom value</td>
</tr>
</tbody>
</table>




<pre>
classmethod set(**kwargs)</pre>
<p>Store an XCom value.
TODO: &#x201C;pickling&#x201D; has been deprecated and JSON is preferred.</p>
<blockquote>
<div>&#x201C;pickling&#x201D; will be removed in Airflow 2.0.</div>
</blockquote>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Returns:</th>
<td class="field-body">None</td>
</tr>
</tbody>
</table>







<pre>
airflow.models.clear_task_instances(tis, session, activate_dag_runs=True, dag=None)</pre>
<p>Clears a set of task instances, but makes sure the running ones
get killed.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>tis</strong> &#x2013; a list of task instances</li>
<li><strong>session</strong> &#x2013; current session</li>
<li><strong>activate_dag_runs</strong> &#x2013; flag to check for active dag run</li>
<li><strong>dag</strong> &#x2013; DAG object</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
airflow.models.get_fernet()</pre>
<p>Deferred load of Fernet key.</p>
<p>This function could fail either because Cryptography is not installed
or because the Fernet key is invalid.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Returns:</th>
<td class="field-body">Fernet object</td>
</tr>
<tr class="field-even field"><th class="field-name">Raises:</th>
<td class="field-body">AirflowException if there&#x2019;s a problem trying to load Fernet</td>
</tr>
</tbody>
</table>



</div>
<div class="section" id="hooks">
<h2 class="sigil_not_in_toc">Hooks</h2>
<p>Hooks are interfaces to external platforms and databases, implementing a common
interface when possible and acting as building blocks for operators.</p>

<pre>
class airflow.hooks.dbapi_hook.DbApiHook(*args, **kwargs)</pre>
<p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.hooks.base_hook.BaseHook</span></code></p>
<p>Abstract base class for sql hooks.</p>

<pre>
bulk_dump(table, tmp_file)</pre>
<p>Dumps a database table into a tab-delimited file</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>table</strong> (<em>str</em>) &#x2013; The name of the source table</li>
<li><strong>tmp_file</strong> (<em>str</em>) &#x2013; The path of the target file</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
bulk_load(table, tmp_file)</pre>
<p>Loads a tab-delimited file into a database table</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>table</strong> (<em>str</em>) &#x2013; The name of the target table</li>
<li><strong>tmp_file</strong> (<em>str</em>) &#x2013; The path of the file to load into the table</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
get_autocommit(conn)</pre>
<p>Get autocommit setting for the provided connection.
Return True if conn.autocommit is set to True.
Return False if conn.autocommit is not set or set to False or conn
does not support autocommit.
:param conn: Connection to get autocommit setting from.
:type conn: connection object.
:return: connection autocommit setting.
:rtype bool.</p>




<pre>
get_conn()</pre>
<p>Returns a connection object</p>




<pre>
get_cursor()</pre>
<p>Returns a cursor</p>




<pre>
get_first(sql, parameters=None)</pre>
<p>Executes the sql and returns the first resulting row.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>sql</strong> (<em>str</em><em> or </em><em>list</em>) &#x2013; the sql statement to be executed (str) or a list of
sql statements to execute</li>
<li><strong>parameters</strong> (<em>mapping</em><em> or </em><em>iterable</em>) &#x2013; The parameters to render the SQL query with.</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
get_pandas_df(sql, parameters=None)</pre>
<p>Executes the sql and returns a pandas dataframe</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>sql</strong> (<em>str</em><em> or </em><em>list</em>) &#x2013; the sql statement to be executed (str) or a list of
sql statements to execute</li>
<li><strong>parameters</strong> (<em>mapping</em><em> or </em><em>iterable</em>) &#x2013; The parameters to render the SQL query with.</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
get_records(sql, parameters=None)</pre>
<p>Executes the sql and returns a set of records.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>sql</strong> (<em>str</em><em> or </em><em>list</em>) &#x2013; the sql statement to be executed (str) or a list of
sql statements to execute</li>
<li><strong>parameters</strong> (<em>mapping</em><em> or </em><em>iterable</em>) &#x2013; The parameters to render the SQL query with.</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
insert_rows(table, rows, target_fields=None, commit_every=1000, replace=False)</pre>
<p>A generic way to insert a set of tuples into a table,
a new transaction is created every commit_every rows</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>table</strong> (<em>str</em>) &#x2013; Name of the target table</li>
<li><strong>rows</strong> (<em>iterable of tuples</em>) &#x2013; The rows to insert into the table</li>
<li><strong>target_fields</strong> (<em>iterable of strings</em>) &#x2013; The names of the columns to fill in the table</li>
<li><strong>commit_every</strong> (<em>int</em>) &#x2013; The maximum number of rows to insert in one
transaction. Set to 0 to insert all rows in one transaction.</li>
<li><strong>replace</strong> (<em>bool</em>) &#x2013; Whether to replace instead of insert</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
run(sql, autocommit=False, parameters=None)</pre>
<p>Runs a command or a list of commands. Pass a list of sql
statements to the sql parameter to get them to execute
sequentially</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>sql</strong> (<em>str</em><em> or </em><em>list</em>) &#x2013; the sql statement to be executed (str) or a list of
sql statements to execute</li>
<li><strong>autocommit</strong> (<em>bool</em>) &#x2013; What to set the connection&#x2019;s autocommit setting to
before executing the query.</li>
<li><strong>parameters</strong> (<em>mapping</em><em> or </em><em>iterable</em>) &#x2013; The parameters to render the SQL query with.</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
set_autocommit(conn, autocommit)</pre>
<p>Sets the autocommit flag on the connection</p>







<pre>
class airflow.hooks.docker_hook.DockerHook(docker_conn_id=&apos;docker_default&apos;, base_url=None, version=None, tls=None)</pre>
<p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.hooks.base_hook.BaseHook</span></code>, <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.utils.log.logging_mixin.LoggingMixin</span></code></p>
<p>Interact with a private Docker registry.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><strong>docker_conn_id</strong> (<em>str</em>) &#x2013; ID of the Airflow connection where
credentials and extra configuration are stored</td>
</tr>
</tbody>
</table>



<span class="target" id="module-airflow.hooks.hive_hooks"></span>
<pre>
class airflow.hooks.hive_hooks.HiveCliHook(hive_cli_conn_id=u&apos;hive_cli_default&apos;, run_as=None, mapred_queue=None, mapred_queue_priority=None, mapred_job_name=None)</pre>
<p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.hooks.base_hook.BaseHook</span></code></p>
<p>Simple wrapper around the hive CLI.</p>
<p>It also supports the <code class="docutils literal notranslate"><span class="pre">beeline</span></code>
a lighter CLI that runs JDBC and is replacing the heavier
traditional CLI. To enable <code class="docutils literal notranslate"><span class="pre">beeline</span></code>, set the use_beeline param in the
extra field of your connection as in <code class="docutils literal notranslate"><span class="pre">{</span> <span class="pre">&quot;use_beeline&quot;:</span> <span class="pre">true</span> <span class="pre">}</span></code></p>
<p>Note that you can also set default hive CLI parameters using the
<code class="docutils literal notranslate"><span class="pre">hive_cli_params</span></code> to be used in your connection as in
<code class="docutils literal notranslate"><span class="pre">{&quot;hive_cli_params&quot;:</span> <span class="pre">&quot;-hiveconf</span> <span class="pre">mapred.job.tracker=some.jobtracker:444&quot;}</span></code>
Parameters passed here can be overridden by run_cli&#x2019;s hive_conf param</p>
<p>The extra connection parameter <code class="docutils literal notranslate"><span class="pre">auth</span></code> gets passed as in the <code class="docutils literal notranslate"><span class="pre">jdbc</span></code>
connection string as is.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>mapred_queue</strong> (<em>string</em>) &#x2013; queue used by the Hadoop Scheduler (Capacity or Fair)</li>
<li><strong>mapred_queue_priority</strong> (<em>string</em>) &#x2013; priority within the job queue.
Possible settings include: VERY_HIGH, HIGH, NORMAL, LOW, VERY_LOW</li>
<li><strong>mapred_job_name</strong> (<em>string</em>) &#x2013; This name will appear in the jobtracker.
This can make monitoring easier.</li>
</ul>
</td>
</tr>
</tbody>
</table>

<pre>
load_df(df, table, field_dict=None, delimiter=u&apos;, &apos;, encoding=u&apos;utf8&apos;, pandas_kwargs=None, **kwargs)</pre>
<p>Loads a pandas DataFrame into hive.</p>
<p>Hive data types will be inferred if not passed but column names will
not be sanitized.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>df</strong> (<em>DataFrame</em>) &#x2013; DataFrame to load into a Hive table</li>
<li><strong>table</strong> (<em>str</em>) &#x2013; target Hive table, use dot notation to target a
specific database</li>
<li><strong>field_dict</strong> (<em>OrderedDict</em>) &#x2013; mapping from column name to hive data type.
Note that it must be OrderedDict so as to keep columns&#x2019; order.</li>
<li><strong>delimiter</strong> (<em>str</em>) &#x2013; field delimiter in the file</li>
<li><strong>encoding</strong> (<em>str</em>) &#x2013; string encoding to use when writing DataFrame to file</li>
<li><strong>pandas_kwargs</strong> (<em>dict</em>) &#x2013; passed to DataFrame.to_csv</li>
<li><strong>kwargs</strong> &#x2013; passed to self.load_file</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
load_file(filepath, table, delimiter=u&apos;, &apos;, field_dict=None, create=True, overwrite=True, partition=None, recreate=False, tblproperties=None)</pre>
<p>Loads a local file into Hive</p>
<p>Note that the table generated in Hive uses <code class="docutils literal notranslate"><span class="pre">STORED</span> <span class="pre">AS</span> <span class="pre">textfile</span></code>
which isn&#x2019;t the most efficient serialization format. If a
large amount of data is loaded and/or if the tables gets
queried considerably, you may want to use this operator only to
stage the data into a temporary table before loading it into its
final destination using a <code class="docutils literal notranslate"><span class="pre">HiveOperator</span></code>.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>filepath</strong> (<em>str</em>) &#x2013; local filepath of the file to load</li>
<li><strong>table</strong> (<em>str</em>) &#x2013; target Hive table, use dot notation to target a
specific database</li>
<li><strong>delimiter</strong> (<em>str</em>) &#x2013; field delimiter in the file</li>
<li><strong>field_dict</strong> (<em>OrderedDict</em>) &#x2013; A dictionary of the fields name in the file
as keys and their Hive types as values.
Note that it must be OrderedDict so as to keep columns&#x2019; order.</li>
<li><strong>create</strong> (<em>bool</em>) &#x2013; whether to create the table if it doesn&#x2019;t exist</li>
<li><strong>overwrite</strong> (<em>bool</em>) &#x2013; whether to overwrite the data in table or partition</li>
<li><strong>partition</strong> (<em>dict</em>) &#x2013; target partition as a dict of partition columns
and values</li>
<li><strong>recreate</strong> (<em>bool</em>) &#x2013; whether to drop and recreate the table at every
execution</li>
<li><strong>tblproperties</strong> (<em>dict</em>) &#x2013; TBLPROPERTIES of the hive table being created</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
run_cli(hql, schema=None, verbose=True, hive_conf=None)</pre>
<p>Run an hql statement using the hive cli. If hive_conf is specified
it should be a dict and the entries will be set as key/value pairs
in HiveConf</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><strong>hive_conf</strong> (<em>dict</em>) &#x2013; if specified these key value pairs will be passed
to hive as <code class="docutils literal notranslate"><span class="pre">-hiveconf</span> <span class="pre">&quot;key&quot;=&quot;value&quot;</span></code>. Note that they will be
passed after the <code class="docutils literal notranslate"><span class="pre">hive_cli_params</span></code> and thus will override
whatever values are specified in the database.</td>
</tr>
</tbody>
</table>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">hh</span> <span class="o">=</span> <span class="n">HiveCliHook</span><span class="p">()</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">result</span> <span class="o">=</span> <span class="n">hh</span><span class="o">.</span><span class="n">run_cli</span><span class="p">(</span><span class="s2">&quot;USE airflow;&quot;</span><span class="p">)</span>
<span class="gp">&gt;&gt;&gt; </span><span class="p">(</span><span class="s2">&quot;OK&quot;</span> <span class="ow">in</span> <span class="n">result</span><span class="p">)</span>
<span class="go">True</span>
</pre>
</div>
</div>




<pre>
test_hql(hql)</pre>
<p>Test an hql statement using the hive cli and EXPLAIN</p>







<pre>
class airflow.hooks.hive_hooks.HiveMetastoreHook(metastore_conn_id=u&apos;metastore_default&apos;)</pre>
<p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.hooks.base_hook.BaseHook</span></code></p>
<p>Wrapper to interact with the Hive Metastore</p>

<pre>
check_for_named_partition(schema, table, partition_name)</pre>
<p>Checks whether a partition with a given name exists</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first simple">
<li><strong>schema</strong> (<em>string</em>) &#x2013; Name of hive schema (database) @table belongs to</li>
<li><strong>table</strong> &#x2013; Name of hive table @partition belongs to</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Partition:</th>
<td class="field-body"><p class="first">Name of the partitions to check for (eg <cite>a=b/c=d</cite>)</p>
</td>
</tr>
<tr class="field-odd field"><th class="field-name">Return type:</th>
<td class="field-body"><p class="first last">boolean</p>
</td>
</tr>
</tbody>
</table>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">hh</span> <span class="o">=</span> <span class="n">HiveMetastoreHook</span><span class="p">()</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">t</span> <span class="o">=</span> <span class="s1">&apos;static_babynames_partitioned&apos;</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">hh</span><span class="o">.</span><span class="n">check_for_named_partition</span><span class="p">(</span><span class="s1">&apos;airflow&apos;</span><span class="p">,</span> <span class="n">t</span><span class="p">,</span> <span class="s2">&quot;ds=2015-01-01&quot;</span><span class="p">)</span>
<span class="go">True</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">hh</span><span class="o">.</span><span class="n">check_for_named_partition</span><span class="p">(</span><span class="s1">&apos;airflow&apos;</span><span class="p">,</span> <span class="n">t</span><span class="p">,</span> <span class="s2">&quot;ds=xxx&quot;</span><span class="p">)</span>
<span class="go">False</span>
</pre>
</div>
</div>




<pre>
check_for_partition(schema, table, partition)</pre>
<p>Checks whether a partition exists</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first simple">
<li><strong>schema</strong> (<em>string</em>) &#x2013; Name of hive schema (database) @table belongs to</li>
<li><strong>table</strong> &#x2013; Name of hive table @partition belongs to</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Partition:</th>
<td class="field-body"><p class="first">Expression that matches the partitions to check for
(eg <cite>a = &#x2018;b&#x2019; AND c = &#x2018;d&#x2019;</cite>)</p>
</td>
</tr>
<tr class="field-odd field"><th class="field-name">Return type:</th>
<td class="field-body"><p class="first last">boolean</p>
</td>
</tr>
</tbody>
</table>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">hh</span> <span class="o">=</span> <span class="n">HiveMetastoreHook</span><span class="p">()</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">t</span> <span class="o">=</span> <span class="s1">&apos;static_babynames_partitioned&apos;</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">hh</span><span class="o">.</span><span class="n">check_for_partition</span><span class="p">(</span><span class="s1">&apos;airflow&apos;</span><span class="p">,</span> <span class="n">t</span><span class="p">,</span> <span class="s2">&quot;ds=&apos;2015-01-01&apos;&quot;</span><span class="p">)</span>
<span class="go">True</span>
</pre>
</div>
</div>




<pre>
get_databases(pattern=u&apos;*&apos;)</pre>
<p>Get a metastore table object</p>




<pre>
get_metastore_client()</pre>
<p>Returns a Hive thrift client.</p>




<pre>
get_partitions(schema, table_name, filter=None)</pre>
<p>Returns a list of all partitions in a table. Works only
for tables with less than 32767 (java short max val).
For subpartitioned table, the number might easily exceed this.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">hh</span> <span class="o">=</span> <span class="n">HiveMetastoreHook</span><span class="p">()</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">t</span> <span class="o">=</span> <span class="s1">&apos;static_babynames_partitioned&apos;</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">parts</span> <span class="o">=</span> <span class="n">hh</span><span class="o">.</span><span class="n">get_partitions</span><span class="p">(</span><span class="n">schema</span><span class="o">=</span><span class="s1">&apos;airflow&apos;</span><span class="p">,</span> <span class="n">table_name</span><span class="o">=</span><span class="n">t</span><span class="p">)</span>
<span class="gp">&gt;&gt;&gt; </span><span class="nb">len</span><span class="p">(</span><span class="n">parts</span><span class="p">)</span>
<span class="go">1</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">parts</span>
<span class="go">[{&apos;ds&apos;: &apos;2015-01-01&apos;}]</span>
</pre>
</div>
</div>




<pre>
get_table(table_name, db=u&apos;default&apos;)</pre>
<p>Get a metastore table object</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">hh</span> <span class="o">=</span> <span class="n">HiveMetastoreHook</span><span class="p">()</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">t</span> <span class="o">=</span> <span class="n">hh</span><span class="o">.</span><span class="n">get_table</span><span class="p">(</span><span class="n">db</span><span class="o">=</span><span class="s1">&apos;airflow&apos;</span><span class="p">,</span> <span class="n">table_name</span><span class="o">=</span><span class="s1">&apos;static_babynames&apos;</span><span class="p">)</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">t</span><span class="o">.</span><span class="n">tableName</span>
<span class="go">&apos;static_babynames&apos;</span>
<span class="gp">&gt;&gt;&gt; </span><span class="p">[</span><span class="n">col</span><span class="o">.</span><span class="n">name</span> <span class="k">for</span> <span class="n">col</span> <span class="ow">in</span> <span class="n">t</span><span class="o">.</span><span class="n">sd</span><span class="o">.</span><span class="n">cols</span><span class="p">]</span>
<span class="go">[&apos;state&apos;, &apos;year&apos;, &apos;name&apos;, &apos;gender&apos;, &apos;num&apos;]</span>
</pre>
</div>
</div>




<pre>
get_tables(db, pattern=u&apos;*&apos;)</pre>
<p>Get a metastore table object</p>




<pre>
max_partition(schema, table_name, field=None, filter_map=None)</pre>
<p>Returns the maximum value for all partitions with given field in a table.
If only one partition key exist in the table, the key will be used as field.
filter_map should be a partition_key:partition_value map and will be used to
filter out partitions.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>schema</strong> (<em>string</em>) &#x2013; schema name.</li>
<li><strong>table_name</strong> (<em>string</em>) &#x2013; table name.</li>
<li><strong>field</strong> (<em>string</em>) &#x2013; partition key to get max partition from.</li>
<li><strong>filter_map</strong> (<em>map</em>) &#x2013; partition_key:partition_value map used for partition filtering.</li>
</ul>
</td>
</tr>
</tbody>
</table>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">hh</span> <span class="o">=</span> <span class="n">HiveMetastoreHook</span><span class="p">()</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">filter_map</span> <span class="o">=</span> <span class="p">{</span><span class="s1">&apos;ds&apos;</span><span class="p">:</span> <span class="s1">&apos;2015-01-01&apos;</span><span class="p">,</span> <span class="s1">&apos;ds&apos;</span><span class="p">:</span> <span class="s1">&apos;2014-01-01&apos;</span><span class="p">}</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">t</span> <span class="o">=</span> <span class="s1">&apos;static_babynames_partitioned&apos;</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">hh</span><span class="o">.</span><span class="n">max_partition</span><span class="p">(</span><span class="n">schema</span><span class="o">=</span><span class="s1">&apos;airflow&apos;</span><span class="p">,</span>        <span class="o">...</span> <span class="n">table_name</span><span class="o">=</span><span class="n">t</span><span class="p">,</span> <span class="n">field</span><span class="o">=</span><span class="s1">&apos;ds&apos;</span><span class="p">,</span> <span class="n">filter_map</span><span class="o">=</span><span class="n">filter_map</span><span class="p">)</span>
<span class="go">&apos;2015-01-01&apos;</span>
</pre>
</div>
</div>




<pre>
table_exists(table_name, db=u&apos;default&apos;)</pre>
<p>Check if table exists</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">hh</span> <span class="o">=</span> <span class="n">HiveMetastoreHook</span><span class="p">()</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">hh</span><span class="o">.</span><span class="n">table_exists</span><span class="p">(</span><span class="n">db</span><span class="o">=</span><span class="s1">&apos;airflow&apos;</span><span class="p">,</span> <span class="n">table_name</span><span class="o">=</span><span class="s1">&apos;static_babynames&apos;</span><span class="p">)</span>
<span class="go">True</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">hh</span><span class="o">.</span><span class="n">table_exists</span><span class="p">(</span><span class="n">db</span><span class="o">=</span><span class="s1">&apos;airflow&apos;</span><span class="p">,</span> <span class="n">table_name</span><span class="o">=</span><span class="s1">&apos;does_not_exist&apos;</span><span class="p">)</span>
<span class="go">False</span>
</pre>
</div>
</div>







<pre>
class airflow.hooks.hive_hooks.HiveServer2Hook(hiveserver2_conn_id=u&apos;hiveserver2_default&apos;)</pre>
<p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.hooks.base_hook.BaseHook</span></code></p>
<p>Wrapper around the pyhive library</p>
<p>Note that the default authMechanism is PLAIN, to override it you
can specify it in the <code class="docutils literal notranslate"><span class="pre">extra</span></code> of your connection in the UI as in</p>

<pre>
get_pandas_df(hql, schema=u&apos;default&apos;)</pre>
<p>Get a pandas dataframe from a Hive query</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">hh</span> <span class="o">=</span> <span class="n">HiveServer2Hook</span><span class="p">()</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">sql</span> <span class="o">=</span> <span class="s2">&quot;SELECT * FROM airflow.static_babynames LIMIT 100&quot;</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">df</span> <span class="o">=</span> <span class="n">hh</span><span class="o">.</span><span class="n">get_pandas_df</span><span class="p">(</span><span class="n">sql</span><span class="p">)</span>
<span class="gp">&gt;&gt;&gt; </span><span class="nb">len</span><span class="p">(</span><span class="n">df</span><span class="o">.</span><span class="n">index</span><span class="p">)</span>
<span class="go">100</span>
</pre>
</div>
</div>




<pre>
get_records(hql, schema=u&apos;default&apos;)</pre>
<p>Get a set of records from a Hive query.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">hh</span> <span class="o">=</span> <span class="n">HiveServer2Hook</span><span class="p">()</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">sql</span> <span class="o">=</span> <span class="s2">&quot;SELECT * FROM airflow.static_babynames LIMIT 100&quot;</span>
<span class="gp">&gt;&gt;&gt; </span><span class="nb">len</span><span class="p">(</span><span class="n">hh</span><span class="o">.</span><span class="n">get_records</span><span class="p">(</span><span class="n">sql</span><span class="p">))</span>
<span class="go">100</span>
</pre>
</div>
</div>







<pre>
class airflow.hooks.http_hook.HttpHook(method=&apos;POST&apos;, http_conn_id=&apos;http_default&apos;)</pre>
<p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.hooks.base_hook.BaseHook</span></code></p>
<p>Interact with HTTP servers.
:param http_conn_id: connection that has the base API url i.e <a class="reference external" href="https://www.google.com/">https://www.google.com/</a></p>
<blockquote>
<div>and optional authentication credentials. Default headers can also be specified in
the Extra field in json format.</div>
</blockquote>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><strong>method</strong> (<em>str</em>) &#x2013; the API method to be called</td>
</tr>
</tbody>
</table>

<pre>
check_response(response)</pre>
<p>Checks the status code and raise an AirflowException exception on non 2XX or 3XX
status codes
:param response: A requests response object
:type response: requests.response</p>




<pre>
get_conn(headers=None)</pre>
<p>Returns http session for use with requests
:param headers: additional headers to be passed through as a dictionary
:type headers: dict</p>




<pre>
run(endpoint, data=None, headers=None, extra_options=None)</pre>
<p>Performs the request
:param endpoint: the endpoint to be called i.e. resource/v1/query?
:type endpoint: str
:param data: payload to be uploaded or request parameters
:type data: dict
:param headers: additional headers to be passed through as a dictionary
:type headers: dict
:param extra_options: additional options to be used when executing the request</p>
<blockquote>
<div>i.e. {&#x2018;check_response&#x2019;: False} to avoid checking raising exceptions on non
2XX or 3XX status codes</div>
</blockquote>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
</tbody>
</table>




<pre>
run_and_check(session, prepped_request, extra_options)</pre>
<p>Grabs extra options like timeout and actually runs the request,
checking for the result
:param session: the session to be used to execute the request
:type session: requests.Session
:param prepped_request: the prepared request generated in run()
:type prepped_request: session.prepare_request
:param extra_options: additional options to be used when executing the request</p>
<blockquote>
<div>i.e. {&#x2018;check_response&#x2019;: False} to avoid checking raising exceptions on non 2XX
or 3XX status codes</div>
</blockquote>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
</tbody>
</table>




<pre>
run_with_advanced_retry(_retry_args, *args, **kwargs)</pre>
<p>Runs Hook.run() with a Tenacity decorator attached to it. This is useful for
connectors which might be disturbed by intermittent issues and should not
instantly fail.
:param _retry_args: Arguments which define the retry behaviour.</p>
<blockquote>
<div>See Tenacity documentation at <a class="reference external" href="https://github.com/jd/tenacity">https://github.com/jd/tenacity</a></div>
</blockquote>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
</tbody>
</table>

<pre>Example: ::</pre>
<p class="first">hook = HttpHook(http_conn_id=&#x2019;my_conn&#x2019;,method=&#x2019;GET&#x2019;)
retry_args = dict(</p>
<blockquote class="last">
<div><blockquote>
<div>wait=tenacity.wait_exponential(),
stop=tenacity.stop_after_attempt(10),
retry=requests.exceptions.ConnectionError</div>
</blockquote>
<p>)
hook.run_with_advanced_retry(</p>
<blockquote>
<div><blockquote>
<div>endpoint=&#x2019;v1/test&#x2019;,
_retry_args=retry_args</div>
</blockquote>
<p>)</p>
</div>
</blockquote>
</div>
</blockquote>









<pre>
class airflow.hooks.druid_hook.DruidDbApiHook(*args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.hooks.dbapi_hook.DbApiHook" title="airflow.hooks.dbapi_hook.DbApiHook"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.hooks.dbapi_hook.DbApiHook</span></code></a></p>
<p>Interact with Druid broker</p>
<p>This hook is purely for users to query druid broker.
For ingestion, please use druidHook.</p>

<pre>
get_conn()</pre>
<p>Establish a connection to druid broker.</p>




<pre>
get_pandas_df(sql, parameters=None)</pre>
<p>Executes the sql and returns a pandas dataframe</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>sql</strong> (<em>str</em><em> or </em><em>list</em>) &#x2013; the sql statement to be executed (str) or a list of
sql statements to execute</li>
<li><strong>parameters</strong> (<em>mapping</em><em> or </em><em>iterable</em>) &#x2013; The parameters to render the SQL query with.</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
get_uri()</pre>
<p>Get the connection uri for druid broker.</p>
<p>e.g: druid://localhost:8082/druid/v2/sql/</p>




<pre>
insert_rows(table, rows, target_fields=None, commit_every=1000)</pre>
<p>A generic way to insert a set of tuples into a table,
a new transaction is created every commit_every rows</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>table</strong> (<em>str</em>) &#x2013; Name of the target table</li>
<li><strong>rows</strong> (<em>iterable of tuples</em>) &#x2013; The rows to insert into the table</li>
<li><strong>target_fields</strong> (<em>iterable of strings</em>) &#x2013; The names of the columns to fill in the table</li>
<li><strong>commit_every</strong> (<em>int</em>) &#x2013; The maximum number of rows to insert in one
transaction. Set to 0 to insert all rows in one transaction.</li>
<li><strong>replace</strong> (<em>bool</em>) &#x2013; Whether to replace instead of insert</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
set_autocommit(conn, autocommit)</pre>
<p>Sets the autocommit flag on the connection</p>







<pre>
class airflow.hooks.druid_hook.DruidHook(druid_ingest_conn_id=&apos;druid_ingest_default&apos;, timeout=1, max_ingestion_time=None)</pre>
<p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.hooks.base_hook.BaseHook</span></code></p>
<p>Connection to Druid overlord for ingestion</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>druid_ingest_conn_id</strong> (<em>string</em>) &#x2013; The connection id to the Druid overlord machine
which accepts index jobs</li>
<li><strong>timeout</strong> (<em>int</em>) &#x2013; The interval between polling
the Druid job for the status of the ingestion job</li>
<li><strong>max_ingestion_time</strong> (<em>int</em>) &#x2013; The maximum ingestion time before assuming the job failed</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.hooks.hdfs_hook.HDFSHook(hdfs_conn_id=&apos;hdfs_default&apos;, proxy_user=None, autoconfig=False)</pre>
<p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.hooks.base_hook.BaseHook</span></code></p>
<p>Interact with HDFS. This class is a wrapper around the snakebite library.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>hdfs_conn_id</strong> &#x2013; Connection id to fetch connection info</li>
<li><strong>proxy_user</strong> (<em>string</em>) &#x2013; effective user for HDFS operations</li>
<li><strong>autoconfig</strong> (<em>bool</em>) &#x2013; use snakebite&#x2019;s automatically configured client</li>
</ul>
</td>
</tr>
</tbody>
</table>

<pre>
get_conn()</pre>
<p>Returns a snakebite HDFSClient object.</p>







<pre>
class airflow.hooks.jdbc_hook.JdbcHook(*args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.hooks.dbapi_hook.DbApiHook" title="airflow.hooks.dbapi_hook.DbApiHook"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.hooks.dbapi_hook.DbApiHook</span></code></a></p>
<p>General hook for jdbc db access.</p>
<p>JDBC URL, username and password will be taken from the predefined connection.
Note that the whole JDBC URL must be specified in the &#x201C;host&#x201D; field in the DB.
Raises an airflow error if the given connection id doesn&#x2019;t exist.</p>

<pre>
get_conn()</pre>
<p>Returns a connection object</p>




<pre>
set_autocommit(conn, autocommit)</pre>
<p>Enable or disable autocommit for the given connection.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><strong>conn</strong> &#x2013; The connection</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th>
<td class="field-body"></td>
</tr>
</tbody>
</table>







<pre>
class airflow.hooks.mssql_hook.MsSqlHook(*args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.hooks.dbapi_hook.DbApiHook" title="airflow.hooks.dbapi_hook.DbApiHook"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.hooks.dbapi_hook.DbApiHook</span></code></a></p>
<p>Interact with Microsoft SQL Server.</p>

<pre>
get_conn()</pre>
<p>Returns a mssql connection object</p>




<pre>
set_autocommit(conn, autocommit)</pre>
<p>Sets the autocommit flag on the connection</p>







<pre>
class airflow.hooks.mysql_hook.MySqlHook(*args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.hooks.dbapi_hook.DbApiHook" title="airflow.hooks.dbapi_hook.DbApiHook"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.hooks.dbapi_hook.DbApiHook</span></code></a></p>
<p>Interact with MySQL.</p>
<p>You can specify charset in the extra field of your connection
as <code class="docutils literal notranslate"><span class="pre">{&quot;charset&quot;:</span> <span class="pre">&quot;utf8&quot;}</span></code>. Also you can choose cursor as
<code class="docutils literal notranslate"><span class="pre">{&quot;cursor&quot;:</span> <span class="pre">&quot;SSCursor&quot;}</span></code>. Refer to the MySQLdb.cursors for more details.</p>

<pre>
bulk_dump(table, tmp_file)</pre>
<p>Dumps a database table into a tab-delimited file</p>




<pre>
bulk_load(table, tmp_file)</pre>
<p>Loads a tab-delimited file into a database table</p>




<pre>
get_autocommit(conn)</pre>
<p>MySql connection gets autocommit in a different way.
:param conn: connection to get autocommit setting from.
:type conn: connection object.
:return: connection autocommit setting
:rtype bool</p>




<pre>
get_conn()</pre>
<p>Returns a mysql connection object</p>




<pre>
set_autocommit(conn, autocommit)</pre>
<p>MySql connection sets autocommit in a different way.</p>







<pre>
class airflow.hooks.oracle_hook.OracleHook(*args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.hooks.dbapi_hook.DbApiHook" title="airflow.hooks.dbapi_hook.DbApiHook"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.hooks.dbapi_hook.DbApiHook</span></code></a></p>
<p>Interact with Oracle SQL.</p>

<pre>
bulk_insert_rows(table, rows, target_fields=None, commit_every=5000)</pre>
<p>A performant bulk insert for cx_Oracle
that uses prepared statements via <cite>executemany()</cite>.
For best performance, pass in <cite>rows</cite> as an iterator.</p>




<pre>
get_conn()</pre>
<p>Returns a oracle connection object
Optional parameters for using a custom DSN connection
(instead of using a server alias from tnsnames.ora)
The dsn (data source name) is the TNS entry
(from the Oracle names server or tnsnames.ora file)
or is a string like the one returned from makedsn().</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>dsn</strong> &#x2013; the host address for the Oracle server</li>
<li><strong>service_name</strong> &#x2013; the db_unique_name of the database
that you are connecting to (CONNECT_DATA part of TNS)</li>
</ul>
</td>
</tr>
</tbody>
</table>
<p>You can set these parameters in the extra fields of your connection
as in <code class="docutils literal notranslate"><span class="pre">{</span> <span class="pre">&quot;dsn&quot;:&quot;some.host.address&quot;</span> <span class="pre">,</span> <span class="pre">&quot;service_name&quot;:&quot;some.service.name&quot;</span> <span class="pre">}</span></code></p>




<pre>
insert_rows(table, rows, target_fields=None, commit_every=1000)</pre>
<p>A generic way to insert a set of tuples into a table,
the whole set of inserts is treated as one transaction
Changes from standard DbApiHook implementation:
- Oracle SQL queries in cx_Oracle can not be terminated with a semicolon (&#x2018;;&#x2019;)
- Replace NaN values with NULL using numpy.nan_to_num (not using is_nan()</p>
<blockquote>
<div>because of input types error for strings)</div>
</blockquote>
<ul class="simple">
<li>Coerce datetime cells to Oracle DATETIME format during insert</li>
</ul>







<pre>
class airflow.hooks.pig_hook.PigCliHook(pig_cli_conn_id=&apos;pig_cli_default&apos;)</pre>
<p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.hooks.base_hook.BaseHook</span></code></p>
<p>Simple wrapper around the pig CLI.</p>
<p>Note that you can also set default pig CLI properties using the
<code class="docutils literal notranslate"><span class="pre">pig_properties</span></code> to be used in your connection as in
<code class="docutils literal notranslate"><span class="pre">{&quot;pig_properties&quot;:</span> <span class="pre">&quot;-Dpig.tmpfilecompression=true&quot;}</span></code></p>

<pre>
run_cli(pig, verbose=True)</pre>
<p>Run an pig script using the pig cli</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">ph</span> <span class="o">=</span> <span class="n">PigCliHook</span><span class="p">()</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">result</span> <span class="o">=</span> <span class="n">ph</span><span class="o">.</span><span class="n">run_cli</span><span class="p">(</span><span class="s2">&quot;ls /;&quot;</span><span class="p">)</span>
<span class="gp">&gt;&gt;&gt; </span><span class="p">(</span><span class="s2">&quot;hdfs://&quot;</span> <span class="ow">in</span> <span class="n">result</span><span class="p">)</span>
<span class="go">True</span>
</pre>
</div>
</div>







<pre>
class airflow.hooks.postgres_hook.PostgresHook(*args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.hooks.dbapi_hook.DbApiHook" title="airflow.hooks.dbapi_hook.DbApiHook"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.hooks.dbapi_hook.DbApiHook</span></code></a></p>
<p>Interact with Postgres.
You can specify ssl parameters in the extra field of your connection
as <code class="docutils literal notranslate"><span class="pre">{&quot;sslmode&quot;:</span> <span class="pre">&quot;require&quot;,</span> <span class="pre">&quot;sslcert&quot;:</span> <span class="pre">&quot;/path/to/cert.pem&quot;,</span> <span class="pre">etc}</span></code>.</p>
<p>Note: For Redshift, use keepalives_idle in the extra connection parameters
and set it to less than 300 seconds.</p>

<pre>
bulk_dump(table, tmp_file)</pre>
<p>Dumps a database table into a tab-delimited file</p>




<pre>
bulk_load(table, tmp_file)</pre>
<p>Loads a tab-delimited file into a database table</p>




<pre>
copy_expert(sql, filename, open=&lt;built-in function open&gt;)</pre>
<p>Executes SQL using psycopg2 copy_expert method.
Necessary to execute COPY command without access to a superuser.</p>
<p>Note: if this method is called with a &#x201C;COPY FROM&#x201D; statement and
the specified input file does not exist, it creates an empty
file and no data is loaded, but the operation succeeds.
So if users want to be aware when the input file does not exist,
they have to check its existence by themselves.</p>




<pre>
get_conn()</pre>
<p>Returns a connection object</p>







<pre>
class airflow.hooks.presto_hook.PrestoHook(*args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.hooks.dbapi_hook.DbApiHook" title="airflow.hooks.dbapi_hook.DbApiHook"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.hooks.dbapi_hook.DbApiHook</span></code></a></p>
<p>Interact with Presto through PyHive!</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">ph</span> <span class="o">=</span> <span class="n">PrestoHook</span><span class="p">()</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">sql</span> <span class="o">=</span> <span class="s2">&quot;SELECT count(1) AS num FROM airflow.static_babynames&quot;</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">ph</span><span class="o">.</span><span class="n">get_records</span><span class="p">(</span><span class="n">sql</span><span class="p">)</span>
<span class="go">[[340698]]</span>
</pre>
</div>
</div>

<pre>
get_conn()</pre>
<p>Returns a connection object</p>




<pre>
get_first(hql, parameters=None)</pre>
<p>Returns only the first row, regardless of how many rows the query
returns.</p>




<pre>
get_pandas_df(hql, parameters=None)</pre>
<p>Get a pandas dataframe from a sql query.</p>




<pre>
get_records(hql, parameters=None)</pre>
<p>Get a set of records from Presto</p>




<pre>
insert_rows(table, rows, target_fields=None)</pre>
<p>A generic way to insert a set of tuples into a table.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>table</strong> (<em>str</em>) &#x2013; Name of the target table</li>
<li><strong>rows</strong> (<em>iterable of tuples</em>) &#x2013; The rows to insert into the table</li>
<li><strong>target_fields</strong> (<em>iterable of strings</em>) &#x2013; The names of the columns to fill in the table</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
run(hql, parameters=None)</pre>
<p>Execute the statement against Presto. Can be used to create views.</p>







<pre>
class airflow.hooks.S3_hook.S3Hook(aws_conn_id=&apos;aws_default&apos;)</pre>
<p>Bases: <a class="reference internal" href="#airflow.contrib.hooks.aws_hook.AwsHook" title="airflow.contrib.hooks.aws_hook.AwsHook"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.contrib.hooks.aws_hook.AwsHook</span></code></a></p>
<p>Interact with AWS S3, using the boto3 library.</p>

<pre>
check_for_bucket(bucket_name)</pre>
<p>Check if bucket_name exists.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><strong>bucket_name</strong> (<em>str</em>) &#x2013; the name of the bucket</td>
</tr>
</tbody>
</table>




<pre>
check_for_key(key, bucket_name=None)</pre>
<p>Checks if a key exists in a bucket</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>key</strong> (<em>str</em>) &#x2013; S3 key that will point to the file</li>
<li><strong>bucket_name</strong> (<em>str</em>) &#x2013; Name of the bucket in which the file is stored</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
check_for_prefix(bucket_name, prefix, delimiter)</pre>
<p>Checks that a prefix exists in a bucket</p>




<pre>
check_for_wildcard_key(wildcard_key, bucket_name=None, delimiter=&apos;&apos;)</pre>
<p>Checks that a key matching a wildcard expression exists in a bucket</p>




<pre>
get_bucket(bucket_name)</pre>
<p>Returns a boto3.S3.Bucket object</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><strong>bucket_name</strong> (<em>str</em>) &#x2013; the name of the bucket</td>
</tr>
</tbody>
</table>




<pre>
get_key(key, bucket_name=None)</pre>
<p>Returns a boto3.s3.Object</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>key</strong> (<em>str</em>) &#x2013; the path to the key</li>
<li><strong>bucket_name</strong> (<em>str</em>) &#x2013; the name of the bucket</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
get_wildcard_key(wildcard_key, bucket_name=None, delimiter=&apos;&apos;)</pre>
<p>Returns a boto3.s3.Object object matching the wildcard expression</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>wildcard_key</strong> (<em>str</em>) &#x2013; the path to the key</li>
<li><strong>bucket_name</strong> (<em>str</em>) &#x2013; the name of the bucket</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
list_keys(bucket_name, prefix=&apos;&apos;, delimiter=&apos;&apos;, page_size=None, max_items=None)</pre>
<p>Lists keys in a bucket under prefix and not containing delimiter</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>bucket_name</strong> (<em>str</em>) &#x2013; the name of the bucket</li>
<li><strong>prefix</strong> (<em>str</em>) &#x2013; a key prefix</li>
<li><strong>delimiter</strong> (<em>str</em>) &#x2013; the delimiter marks key hierarchy.</li>
<li><strong>page_size</strong> (<em>int</em>) &#x2013; pagination size</li>
<li><strong>max_items</strong> (<em>int</em>) &#x2013; maximum items to return</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
list_prefixes(bucket_name, prefix=&apos;&apos;, delimiter=&apos;&apos;, page_size=None, max_items=None)</pre>
<p>Lists prefixes in a bucket under prefix</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>bucket_name</strong> (<em>str</em>) &#x2013; the name of the bucket</li>
<li><strong>prefix</strong> (<em>str</em>) &#x2013; a key prefix</li>
<li><strong>delimiter</strong> (<em>str</em>) &#x2013; the delimiter marks key hierarchy.</li>
<li><strong>page_size</strong> (<em>int</em>) &#x2013; pagination size</li>
<li><strong>max_items</strong> (<em>int</em>) &#x2013; maximum items to return</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
load_bytes(bytes_data, key, bucket_name=None, replace=False, encrypt=False)</pre>
<p>Loads bytes to S3</p>
<p>This is provided as a convenience to drop a string in S3. It uses the
boto infrastructure to ship a file to s3.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>bytes_data</strong> (<em>bytes</em>) &#x2013; bytes to set as content for the key.</li>
<li><strong>key</strong> (<em>str</em>) &#x2013; S3 key that will point to the file</li>
<li><strong>bucket_name</strong> (<em>str</em>) &#x2013; Name of the bucket in which to store the file</li>
<li><strong>replace</strong> (<em>bool</em>) &#x2013; A flag to decide whether or not to overwrite the key
if it already exists</li>
<li><strong>encrypt</strong> (<em>bool</em>) &#x2013; If True, the file will be encrypted on the server-side
by S3 and will be stored in an encrypted form while at rest in S3.</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
load_file(filename, key, bucket_name=None, replace=False, encrypt=False)</pre>
<p>Loads a local file to S3</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>filename</strong> (<em>str</em>) &#x2013; name of the file to load.</li>
<li><strong>key</strong> (<em>str</em>) &#x2013; S3 key that will point to the file</li>
<li><strong>bucket_name</strong> (<em>str</em>) &#x2013; Name of the bucket in which to store the file</li>
<li><strong>replace</strong> (<em>bool</em>) &#x2013; A flag to decide whether or not to overwrite the key
if it already exists. If replace is False and the key exists, an
error will be raised.</li>
<li><strong>encrypt</strong> (<em>bool</em>) &#x2013; If True, the file will be encrypted on the server-side
by S3 and will be stored in an encrypted form while at rest in S3.</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
load_string(string_data, key, bucket_name=None, replace=False, encrypt=False, encoding=&apos;utf-8&apos;)</pre>
<p>Loads a string to S3</p>
<p>This is provided as a convenience to drop a string in S3. It uses the
boto infrastructure to ship a file to s3.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>string_data</strong> (<em>str</em>) &#x2013; string to set as content for the key.</li>
<li><strong>key</strong> (<em>str</em>) &#x2013; S3 key that will point to the file</li>
<li><strong>bucket_name</strong> (<em>str</em>) &#x2013; Name of the bucket in which to store the file</li>
<li><strong>replace</strong> (<em>bool</em>) &#x2013; A flag to decide whether or not to overwrite the key
if it already exists</li>
<li><strong>encrypt</strong> (<em>bool</em>) &#x2013; If True, the file will be encrypted on the server-side
by S3 and will be stored in an encrypted form while at rest in S3.</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
read_key(key, bucket_name=None)</pre>
<p>Reads a key from S3</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>key</strong> (<em>str</em>) &#x2013; S3 key that will point to the file</li>
<li><strong>bucket_name</strong> (<em>str</em>) &#x2013; Name of the bucket in which the file is stored</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
select_key(key, bucket_name=None, expression=&apos;SELECT * FROM S3Object&apos;, expression_type=&apos;SQL&apos;, input_serialization={&apos;CSV&apos;: {}}, output_serialization={&apos;CSV&apos;: {}})</pre>
<p>Reads a key with S3 Select.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first simple">
<li><strong>key</strong> (<em>str</em>) &#x2013; S3 key that will point to the file</li>
<li><strong>bucket_name</strong> (<em>str</em>) &#x2013; Name of the bucket in which the file is stored</li>
<li><strong>expression</strong> (<em>str</em>) &#x2013; S3 Select expression</li>
<li><strong>expression_type</strong> (<em>str</em>) &#x2013; S3 Select expression type</li>
<li><strong>input_serialization</strong> (<em>dict</em>) &#x2013; S3 Select input data serialization format</li>
<li><strong>output_serialization</strong> (<em>dict</em>) &#x2013; S3 Select output data serialization format</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th>
<td class="field-body"><p class="first">retrieved subset of original data by S3 Select</p>
</td>
</tr>
<tr class="field-odd field"><th class="field-name">Return type:</th>
<td class="field-body"><p class="first last">str</p>
</td>
</tr>
</tbody>
</table>
<div class="admonition seealso">
<p class="first admonition-title">See also</p>
<p class="last">For more details about S3 Select parameters:
<a class="reference external" href="http://boto3.readthedocs.io/en/latest/reference/services/s3.html#S3.Client.select_object_content">http://boto3.readthedocs.io/en/latest/reference/services/s3.html#S3.Client.select_object_content</a></p>
</div>







<pre>
class airflow.hooks.samba_hook.SambaHook(samba_conn_id)</pre>
<p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.hooks.base_hook.BaseHook</span></code></p>
<p>Allows for interaction with an samba server.</p>




<pre>
class airflow.hooks.slack_hook.SlackHook(token=None, slack_conn_id=None)</pre>
<p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.hooks.base_hook.BaseHook</span></code></p>
<p>Interact with Slack, using slackclient library.</p>




<pre>
class airflow.hooks.sqlite_hook.SqliteHook(*args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.hooks.dbapi_hook.DbApiHook" title="airflow.hooks.dbapi_hook.DbApiHook"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.hooks.dbapi_hook.DbApiHook</span></code></a></p>
<p>Interact with SQLite.</p>

<pre>
get_conn()</pre>
<p>Returns a sqlite connection object</p>







<pre>
class airflow.hooks.webhdfs_hook.WebHDFSHook(webhdfs_conn_id=&apos;webhdfs_default&apos;, proxy_user=None)</pre>
<p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.hooks.base_hook.BaseHook</span></code></p>
<p>Interact with HDFS. This class is a wrapper around the hdfscli library.</p>

<pre>
check_for_path(hdfs_path)</pre>
<p>Check for the existence of a path in HDFS by querying FileStatus.</p>




<pre>
get_conn()</pre>
<p>Returns a hdfscli InsecureClient object.</p>




<pre>
load_file(source, destination, overwrite=True, parallelism=1, **kwargs)</pre>
<p>Uploads a file to HDFS</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>source</strong> (<em>str</em>) &#x2013; Local path to file or folder. If a folder, all the files
inside of it will be uploaded (note that this implies that folders empty
of files will not be created remotely).</li>
<li><strong>destination</strong> (<em>str</em>) &#x2013; PTarget HDFS path. If it already exists and is a
directory, files will be uploaded inside.</li>
<li><strong>overwrite</strong> (<em>bool</em>) &#x2013; Overwrite any existing file or directory.</li>
<li><strong>parallelism</strong> (<em>int</em>) &#x2013; Number of threads to use for parallelization. A value of
<cite>0</cite> (or negative) uses as many threads as there are files.</li>
<li><strong>**kwargs</strong> &#x2013; Keyword arguments forwarded to <code class="xref py py-meth docutils literal notranslate"><span class="pre">upload()</span></code>.</li>
</ul>
</td>
</tr>
</tbody>
</table>







<pre>
class airflow.hooks.zendesk_hook.ZendeskHook(zendesk_conn_id)</pre>
<p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.hooks.base_hook.BaseHook</span></code></p>
<p>A hook to talk to Zendesk</p>

<pre>
call(path, query=None, get_all_pages=True, side_loading=False)</pre>
<p>Call Zendesk API and return results</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>path</strong> &#x2013; The Zendesk API to call</li>
<li><strong>query</strong> &#x2013; Query parameters</li>
<li><strong>get_all_pages</strong> &#x2013; Accumulate results over all pages before
returning. Due to strict rate limiting, this can often timeout.
Waits for recommended period between tries after a timeout.</li>
<li><strong>side_loading</strong> &#x2013; Retrieve related records as part of a single
request. In order to enable side-loading, add an &#x2018;include&#x2019;
query parameter containing a comma-separated list of resources
to load. For more information on side-loading see
<a class="reference external" href="https://developer.zendesk.com/rest_api/docs/core/side_loading">https://developer.zendesk.com/rest_api/docs/core/side_loading</a></li>
</ul>
</td>
</tr>
</tbody>
</table>






<div class="section" id="community-contributed-hooks">
<h3 class="sigil_not_in_toc">Community contributed hooks</h3>

<pre>
class airflow.contrib.hooks.aws_dynamodb_hook.AwsDynamoDBHook(table_keys=None, table_name=None, region_name=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.contrib.hooks.aws_hook.AwsHook" title="airflow.contrib.hooks.aws_hook.AwsHook"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.contrib.hooks.aws_hook.AwsHook</span></code></a></p>
<p>Interact with AWS DynamoDB.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>table_keys</strong> (<em>list</em>) &#x2013; partition key and sort key</li>
<li><strong>table_name</strong> (<em>str</em>) &#x2013; target DynamoDB table</li>
<li><strong>region_name</strong> (<em>str</em>) &#x2013; aws region name (example: us-east-1)</li>
</ul>
</td>
</tr>
</tbody>
</table>

<pre>
write_batch_data(items)</pre>
<p>Write batch items to dynamodb table with provisioned throughout capacity.</p>







<pre>
class airflow.contrib.hooks.aws_hook.AwsHook(aws_conn_id=&apos;aws_default&apos;)</pre>
<p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.hooks.base_hook.BaseHook</span></code></p>
<p>Interact with AWS.
This class is a thin wrapper around the boto3 python library.</p>

<pre>
get_credentials(region_name=None)</pre>
<p>Get the underlying <cite>botocore.Credentials</cite> object.</p>
<p>This contains the attributes: access_key, secret_key and token.</p>




<pre>
get_session(region_name=None)</pre>
<p>Get the underlying boto3.session.</p>







<pre>
class airflow.contrib.hooks.aws_lambda_hook.AwsLambdaHook(function_name, region_name=None, log_type=&apos;None&apos;, qualifier=&apos;$LATEST&apos;, invocation_type=&apos;RequestResponse&apos;, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.contrib.hooks.aws_hook.AwsHook" title="airflow.contrib.hooks.aws_hook.AwsHook"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.contrib.hooks.aws_hook.AwsHook</span></code></a></p>
<p>Interact with AWS Lambda</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>function_name</strong> (<em>str</em>) &#x2013; AWS Lambda Function Name</li>
<li><strong>region_name</strong> (<em>str</em>) &#x2013; AWS Region Name (example: us-west-2)</li>
<li><strong>log_type</strong> (<em>str</em>) &#x2013; Tail Invocation Request</li>
<li><strong>qualifier</strong> (<em>str</em>) &#x2013; AWS Lambda Function Version or Alias Name</li>
<li><strong>invocation_type</strong> (<em>str</em>) &#x2013; AWS Lambda Invocation Type (RequestResponse, Event etc)</li>
</ul>
</td>
</tr>
</tbody>
</table>

<pre>
invoke_lambda(payload)</pre>
<p>Invoke Lambda Function</p>







<pre>
class airflow.contrib.hooks.azure_data_lake_hook.AzureDataLakeHook(azure_data_lake_conn_id=&apos;azure_data_lake_default&apos;)</pre>
<p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.hooks.base_hook.BaseHook</span></code></p>
<p>Interacts with Azure Data Lake.</p>
<p>Client ID and client secret should be in user and password parameters.
Tenant and account name should be extra field as
{&#x201C;tenant&#x201D;: &#x201C;&lt;TENANT&gt;&#x201D;, &#x201C;account_name&#x201D;: &#x201C;ACCOUNT_NAME&#x201D;}.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><strong>azure_data_lake_conn_id</strong> (<em>str</em>) &#x2013; Reference to the Azure Data Lake connection.</td>
</tr>
</tbody>
</table>

<pre>
check_for_file(file_path)</pre>
<p>Check if a file exists on Azure Data Lake.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><strong>file_path</strong> (<em>str</em>) &#x2013; Path and name of the file.</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th>
<td class="field-body">True if the file exists, False otherwise.</td>
</tr>
</tbody>
</table>
<p>:rtype bool</p>




<pre>
download_file(local_path, remote_path, nthreads=64, overwrite=True, buffersize=4194304, blocksize=4194304)</pre>
<p>Download a file from Azure Blob Storage.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>local_path</strong> (<em>str</em>) &#x2013; local path. If downloading a single file, will write to this
specific file, unless it is an existing directory, in which case a file is
created within it. If downloading multiple files, this is the root
directory to write within. Will create directories as required.</li>
<li><strong>remote_path</strong> (<em>str</em>) &#x2013; remote path/globstring to use to find remote files.
Recursive glob patterns using <cite>**</cite> are not supported.</li>
<li><strong>nthreads</strong> (<em>int</em>) &#x2013; Number of threads to use. If None, uses the number of cores.</li>
<li><strong>overwrite</strong> (<em>bool</em>) &#x2013; Whether to forcibly overwrite existing files/directories.
If False and remote path is a directory, will quit regardless if any files
would be overwritten or not. If True, only matching filenames are actually
overwritten.</li>
<li><strong>buffersize</strong> (<em>int</em>) &#x2013; int [2**22]
Number of bytes for internal buffer. This block cannot be bigger than
a chunk and cannot be smaller than a block.</li>
<li><strong>blocksize</strong> (<em>int</em>) &#x2013; int [2**22]
Number of bytes for a block. Within each chunk, we write a smaller
block for each API call. This block cannot be bigger than a chunk.</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
get_conn()</pre>
<p>Return a AzureDLFileSystem object.</p>




<pre>
upload_file(local_path, remote_path, nthreads=64, overwrite=True, buffersize=4194304, blocksize=4194304)</pre>
<p>Upload a file to Azure Data Lake.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>local_path</strong> (<em>str</em>) &#x2013; local path. Can be single file, directory (in which case,
upload recursively) or glob pattern. Recursive glob patterns using <cite>**</cite>
are not supported.</li>
<li><strong>remote_path</strong> (<em>str</em>) &#x2013; Remote path to upload to; if multiple files, this is the
dircetory root to write within.</li>
<li><strong>nthreads</strong> (<em>int</em>) &#x2013; Number of threads to use. If None, uses the number of cores.</li>
<li><strong>overwrite</strong> (<em>bool</em>) &#x2013; Whether to forcibly overwrite existing files/directories.
If False and remote path is a directory, will quit regardless if any files
would be overwritten or not. If True, only matching filenames are actually
overwritten.</li>
<li><strong>buffersize</strong> (<em>int</em>) &#x2013; int [2**22]
Number of bytes for internal buffer. This block cannot be bigger than
a chunk and cannot be smaller than a block.</li>
<li><strong>blocksize</strong> (<em>int</em>) &#x2013; int [2**22]
Number of bytes for a block. Within each chunk, we write a smaller
block for each API call. This block cannot be bigger than a chunk.</li>
</ul>
</td>
</tr>
</tbody>
</table>







<pre>
class airflow.contrib.hooks.azure_fileshare_hook.AzureFileShareHook(wasb_conn_id=&apos;wasb_default&apos;)</pre>
<p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.hooks.base_hook.BaseHook</span></code></p>
<p>Interacts with Azure FileShare Storage.</p>
<p>Additional options passed in the &#x2018;extra&#x2019; field of the connection will be
passed to the <cite>FileService()</cite> constructor.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><strong>wasb_conn_id</strong> (<em>str</em>) &#x2013; Reference to the wasb connection.</td>
</tr>
</tbody>
</table>

<pre>
check_for_directory(share_name, directory_name, **kwargs)</pre>
<p>Check if a directory exists on Azure File Share.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first simple">
<li><strong>share_name</strong> (<em>str</em>) &#x2013; Name of the share.</li>
<li><strong>directory_name</strong> (<em>str</em>) &#x2013; Name of the directory.</li>
<li><strong>kwargs</strong> (<em>object</em>) &#x2013; Optional keyword arguments that
<cite>FileService.exists()</cite> takes.</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th>
<td class="field-body"><p class="first last">True if the file exists, False otherwise.</p>
</td>
</tr>
</tbody>
</table>
<p>:rtype bool</p>




<pre>
check_for_file(share_name, directory_name, file_name, **kwargs)</pre>
<p>Check if a file exists on Azure File Share.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first simple">
<li><strong>share_name</strong> (<em>str</em>) &#x2013; Name of the share.</li>
<li><strong>directory_name</strong> (<em>str</em>) &#x2013; Name of the directory.</li>
<li><strong>file_name</strong> (<em>str</em>) &#x2013; Name of the file.</li>
<li><strong>kwargs</strong> (<em>object</em>) &#x2013; Optional keyword arguments that
<cite>FileService.exists()</cite> takes.</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th>
<td class="field-body"><p class="first last">True if the file exists, False otherwise.</p>
</td>
</tr>
</tbody>
</table>
<p>:rtype bool</p>




<pre>
create_directory(share_name, directory_name, **kwargs)</pre>
<p>Create a new direcotry on a Azure File Share.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first simple">
<li><strong>share_name</strong> (<em>str</em>) &#x2013; Name of the share.</li>
<li><strong>directory_name</strong> (<em>str</em>) &#x2013; Name of the directory.</li>
<li><strong>kwargs</strong> (<em>object</em>) &#x2013; Optional keyword arguments that
<cite>FileService.create_directory()</cite> takes.</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th>
<td class="field-body"><p class="first last">A list of files and directories</p>
</td>
</tr>
</tbody>
</table>
<p>:rtype list</p>




<pre>
get_conn()</pre>
<p>Return the FileService object.</p>




<pre>
get_file(file_path, share_name, directory_name, file_name, **kwargs)</pre>
<p>Download a file from Azure File Share.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>file_path</strong> (<em>str</em>) &#x2013; Where to store the file.</li>
<li><strong>share_name</strong> (<em>str</em>) &#x2013; Name of the share.</li>
<li><strong>directory_name</strong> (<em>str</em>) &#x2013; Name of the directory.</li>
<li><strong>file_name</strong> (<em>str</em>) &#x2013; Name of the file.</li>
<li><strong>kwargs</strong> (<em>object</em>) &#x2013; Optional keyword arguments that
<cite>FileService.get_file_to_path()</cite> takes.</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
get_file_to_stream(stream, share_name, directory_name, file_name, **kwargs)</pre>
<p>Download a file from Azure File Share.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>stream</strong> (<em>file-like object</em>) &#x2013; A filehandle to store the file to.</li>
<li><strong>share_name</strong> (<em>str</em>) &#x2013; Name of the share.</li>
<li><strong>directory_name</strong> (<em>str</em>) &#x2013; Name of the directory.</li>
<li><strong>file_name</strong> (<em>str</em>) &#x2013; Name of the file.</li>
<li><strong>kwargs</strong> (<em>object</em>) &#x2013; Optional keyword arguments that
<cite>FileService.get_file_to_stream()</cite> takes.</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
list_directories_and_files(share_name, directory_name=None, **kwargs)</pre>
<p>Return the list of directories and files stored on a Azure File Share.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first simple">
<li><strong>share_name</strong> (<em>str</em>) &#x2013; Name of the share.</li>
<li><strong>directory_name</strong> (<em>str</em>) &#x2013; Name of the directory.</li>
<li><strong>kwargs</strong> (<em>object</em>) &#x2013; Optional keyword arguments that
<cite>FileService.list_directories_and_files()</cite> takes.</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th>
<td class="field-body"><p class="first last">A list of files and directories</p>
</td>
</tr>
</tbody>
</table>
<p>:rtype list</p>




<pre>
load_file(file_path, share_name, directory_name, file_name, **kwargs)</pre>
<p>Upload a file to Azure File Share.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>file_path</strong> (<em>str</em>) &#x2013; Path to the file to load.</li>
<li><strong>share_name</strong> (<em>str</em>) &#x2013; Name of the share.</li>
<li><strong>directory_name</strong> (<em>str</em>) &#x2013; Name of the directory.</li>
<li><strong>file_name</strong> (<em>str</em>) &#x2013; Name of the file.</li>
<li><strong>kwargs</strong> (<em>object</em>) &#x2013; Optional keyword arguments that
<cite>FileService.create_file_from_path()</cite> takes.</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
load_stream(stream, share_name, directory_name, file_name, count, **kwargs)</pre>
<p>Upload a stream to Azure File Share.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>stream</strong> (<em>file-like</em>) &#x2013; Opened file/stream to upload as the file content.</li>
<li><strong>share_name</strong> (<em>str</em>) &#x2013; Name of the share.</li>
<li><strong>directory_name</strong> (<em>str</em>) &#x2013; Name of the directory.</li>
<li><strong>file_name</strong> (<em>str</em>) &#x2013; Name of the file.</li>
<li><strong>count</strong> (<em>int</em>) &#x2013; Size of the stream in bytes</li>
<li><strong>kwargs</strong> (<em>object</em>) &#x2013; Optional keyword arguments that
<cite>FileService.create_file_from_stream()</cite> takes.</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
load_string(string_data, share_name, directory_name, file_name, **kwargs)</pre>
<p>Upload a string to Azure File Share.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>string_data</strong> (<em>str</em>) &#x2013; String to load.</li>
<li><strong>share_name</strong> (<em>str</em>) &#x2013; Name of the share.</li>
<li><strong>directory_name</strong> (<em>str</em>) &#x2013; Name of the directory.</li>
<li><strong>file_name</strong> (<em>str</em>) &#x2013; Name of the file.</li>
<li><strong>kwargs</strong> (<em>object</em>) &#x2013; Optional keyword arguments that
<cite>FileService.create_file_from_text()</cite> takes.</li>
</ul>
</td>
</tr>
</tbody>
</table>







<pre>
class airflow.contrib.hooks.bigquery_hook.BigQueryHook(bigquery_conn_id=&apos;bigquery_default&apos;, delegate_to=None, use_legacy_sql=True)</pre>
<p>Bases: <a class="reference internal" href="#airflow.contrib.hooks.gcp_api_base_hook.GoogleCloudBaseHook" title="airflow.contrib.hooks.gcp_api_base_hook.GoogleCloudBaseHook"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.contrib.hooks.gcp_api_base_hook.GoogleCloudBaseHook</span></code></a>, <a class="reference internal" href="#airflow.hooks.dbapi_hook.DbApiHook" title="airflow.hooks.dbapi_hook.DbApiHook"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.hooks.dbapi_hook.DbApiHook</span></code></a>, <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.utils.log.logging_mixin.LoggingMixin</span></code></p>
<p>Interact with BigQuery. This hook uses the Google Cloud Platform
connection.</p>

<pre>
get_conn()</pre>
<p>Returns a BigQuery PEP 249 connection object.</p>




<pre>
get_pandas_df(sql, parameters=None, dialect=None)</pre>
<p>Returns a Pandas DataFrame for the results produced by a BigQuery
query. The DbApiHook method must be overridden because Pandas
doesn&#x2019;t support PEP 249 connections, except for SQLite. See:</p>
<p><a class="reference external" href="https://github.com/pydata/pandas/blob/master/pandas/io/sql.py#L447">https://github.com/pydata/pandas/blob/master/pandas/io/sql.py#L447</a>
<a class="reference external" href="https://github.com/pydata/pandas/issues/6900">https://github.com/pydata/pandas/issues/6900</a></p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>sql</strong> (<em>string</em>) &#x2013; The BigQuery SQL to execute.</li>
<li><strong>parameters</strong> (<em>mapping</em><em> or </em><em>iterable</em>) &#x2013; The parameters to render the SQL query with (not
used, leave to override superclass method)</li>
<li><strong>dialect</strong> (<em>string in {&apos;legacy&apos;</em><em>, </em><em>&apos;standard&apos;}</em>) &#x2013; Dialect of BigQuery SQL &#x2013; legacy SQL or standard SQL
defaults to use <cite>self.use_legacy_sql</cite> if not specified</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
get_service()</pre>
<p>Returns a BigQuery service object.</p>




<pre>
insert_rows(table, rows, target_fields=None, commit_every=1000)</pre>
<p>Insertion is currently unsupported. Theoretically, you could use
BigQuery&#x2019;s streaming API to insert rows into a table, but this hasn&#x2019;t
been implemented.</p>




<pre>
table_exists(project_id, dataset_id, table_id)</pre>
<p>Checks for the existence of a table in Google BigQuery.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>project_id</strong> (<em>string</em>) &#x2013; The Google cloud project in which to look for the
table. The connection supplied to the hook must provide access to
the specified project.</li>
<li><strong>dataset_id</strong> (<em>string</em>) &#x2013; The name of the dataset in which to look for the
table.</li>
<li><strong>table_id</strong> (<em>string</em>) &#x2013; The name of the table to check the existence of.</li>
</ul>
</td>
</tr>
</tbody>
</table>







<pre>
class airflow.contrib.hooks.cassandra_hook.CassandraHook(cassandra_conn_id=&apos;cassandra_default&apos;)</pre>
<p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.hooks.base_hook.BaseHook</span></code>, <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.utils.log.logging_mixin.LoggingMixin</span></code></p>
<p>Hook used to interact with Cassandra</p>
<p>Contact points can be specified as a comma-separated string in the &#x2018;hosts&#x2019;
field of the connection.</p>
<p>Port can be specified in the port field of the connection.</p>
<p>If SSL is enabled in Cassandra, pass in a dict in the extra field as kwargs for
<code class="docutils literal notranslate"><span class="pre">ssl.wrap_socket()</span></code>. For example:</p>
<blockquote>
<div>
<pre>{</pre>

<pre>&#x2018;ssl_options&#x2019; : {</pre>
&#x2018;ca_certs&#x2019; : PATH_TO_CA_CERTS

<p class="last">}</p>


<p>}</p>
</div>
</blockquote>

<pre>Default load balancing policy is RoundRobinPolicy. To specify a different LB policy:</pre>
<ul class="first last">
<li>
<pre>DCAwareRoundRobinPolicy</pre>

<pre>{</pre>
<blockquote class="first">
<div><p>&#x2018;load_balancing_policy&#x2019;: &#x2018;DCAwareRoundRobinPolicy&#x2019;,
&#x2018;load_balancing_policy_args&#x2019;: {</p>
<blockquote>
<div><p>&#x2018;local_dc&#x2019;: LOCAL_DC_NAME,                      // optional
&#x2018;used_hosts_per_remote_dc&#x2019;: SOME_INT_VALUE,     // optional</p>
</div>
</blockquote>
<p>}</p>
</div>
</blockquote>
<p class="last">}</p>




</li>
<li>
<pre>WhiteListRoundRobinPolicy</pre>

<pre>{</pre>
<p class="first">&#x2018;load_balancing_policy&#x2019;: &#x2018;WhiteListRoundRobinPolicy&#x2019;,
&#x2018;load_balancing_policy_args&#x2019;: {</p>
<blockquote>
<div><p>&#x2018;hosts&#x2019;: [&#x2018;HOST1&#x2019;, &#x2018;HOST2&#x2019;, &#x2018;HOST3&#x2019;]</p>
</div>
</blockquote>
<p class="last">}</p>


<p class="last">}</p>


</li>
<li>
<pre>TokenAwarePolicy</pre>

<pre>{</pre>
<p class="first">&#x2018;load_balancing_policy&#x2019;: &#x2018;TokenAwarePolicy&#x2019;,
&#x2018;load_balancing_policy_args&#x2019;: {</p>
<blockquote>
<div><p>&#x2018;child_load_balancing_policy&#x2019;: CHILD_POLICY_NAME, // optional
&#x2018;child_load_balancing_policy_args&#x2019;: { &#x2026; }       // optional</p>
</div>
</blockquote>
<p class="last">}</p>


<p class="last">}</p>


</li>
</ul>


<p>For details of the Cluster config, see cassandra.cluster.</p>

<pre>
get_conn()</pre>
<p>Returns a cassandra Session object</p>




<pre>
record_exists(table, keys)</pre>
<p>Checks if a record exists in Cassandra</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>table</strong> (<em>string</em>) &#x2013; Target Cassandra table.
Use dot notation to target a specific keyspace.</li>
<li><strong>keys</strong> (<em>dict</em>) &#x2013; The keys and their values to check the existence.</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
shutdown_cluster()</pre>
<p>Closes all sessions and connections associated with this Cluster.</p>







<pre>
class airflow.contrib.hooks.cloudant_hook.CloudantHook(cloudant_conn_id=&apos;cloudant_default&apos;)</pre>
<p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.hooks.base_hook.BaseHook</span></code></p>
<p>Interact with Cloudant.</p>
<p>This class is a thin wrapper around the cloudant python library. See the
documentation <a class="reference external" href="https://github.com/cloudant-labs/cloudant-python">here</a>.</p>

<pre>
db()</pre>
<p>Returns the Database object for this hook.</p>
<p>See the documentation for cloudant-python here
<a class="reference external" href="https://github.com/cloudant-labs/cloudant-python">https://github.com/cloudant-labs/cloudant-python</a>.</p>







<pre>
class airflow.contrib.hooks.databricks_hook.DatabricksHook(databricks_conn_id=&apos;databricks_default&apos;, timeout_seconds=180, retry_limit=3)</pre>
<p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.hooks.base_hook.BaseHook</span></code>, <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.utils.log.logging_mixin.LoggingMixin</span></code></p>
<p>Interact with Databricks.</p>

<pre>
submit_run(json)</pre>
<p>Utility function to call the <code class="docutils literal notranslate"><span class="pre">api/2.0/jobs/runs/submit</span></code> endpoint.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><strong>json</strong> (<em>dict</em>) &#x2013; The data used in the body of the request to the <code class="docutils literal notranslate"><span class="pre">submit</span></code> endpoint.</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th>
<td class="field-body">the run_id as a string</td>
</tr>
<tr class="field-odd field"><th class="field-name">Return type:</th>
<td class="field-body">string</td>
</tr>
</tbody>
</table>







<pre>
class airflow.contrib.hooks.datadog_hook.DatadogHook(datadog_conn_id=&apos;datadog_default&apos;)</pre>
<p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.hooks.base_hook.BaseHook</span></code>, <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.utils.log.logging_mixin.LoggingMixin</span></code></p>
<p>Uses datadog API to send metrics of practically anything measurable,
so it&#x2019;s possible to track # of db records inserted/deleted, records read
from file and many other useful metrics.</p>
<p>Depends on the datadog API, which has to be deployed on the same server where
Airflow runs.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>datadog_conn_id</strong> &#x2013; The connection to datadog, containing metadata for api keys.</li>
<li><strong>datadog_conn_id</strong> &#x2013; string</li>
</ul>
</td>
</tr>
</tbody>
</table>

<pre>
post_event(title, text, tags=None, alert_type=None, aggregation_key=None)</pre>
<p>Posts an event to datadog (processing finished, potentially alerts, other issues)
Think about this as a means to maintain persistence of alerts, rather than
alerting itself.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>title</strong> (<em>string</em>) &#x2013; The title of the event</li>
<li><strong>text</strong> (<em>string</em>) &#x2013; The body of the event (more information)</li>
<li><strong>tags</strong> (<em>list</em>) &#x2013; List of string tags to apply to the event</li>
<li><strong>alert_type</strong> (<em>string</em>) &#x2013; The alert type for the event, one of
[&#x201C;error&#x201D;, &#x201C;warning&#x201D;, &#x201C;info&#x201D;, &#x201C;success&#x201D;]</li>
<li><strong>aggregation_key</strong> (<em>string</em>) &#x2013; Key that can be used to aggregate this event in a stream</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
query_metric(query, from_seconds_ago, to_seconds_ago)</pre>
<p>Queries datadog for a specific metric, potentially with some
function applied to it and returns the results.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>query</strong> (<em>string</em>) &#x2013; The datadog query to execute (see datadog docs)</li>
<li><strong>from_seconds_ago</strong> (<em>int</em>) &#x2013; How many seconds ago to start querying for.</li>
<li><strong>to_seconds_ago</strong> (<em>int</em>) &#x2013; Up to how many seconds ago to query for.</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
send_metric(metric_name, datapoint, tags=None)</pre>
<p>Sends a single datapoint metric to DataDog</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>metric_name</strong> (<em>string</em>) &#x2013; The name of the metric</li>
<li><strong>datapoint</strong> (<em>integer</em><em> or </em><em>float</em>) &#x2013; A single integer or float related to the metric</li>
<li><strong>tags</strong> (<em>list</em>) &#x2013; A list of tags associated with the metric</li>
</ul>
</td>
</tr>
</tbody>
</table>







<pre>
class airflow.contrib.hooks.datastore_hook.DatastoreHook(datastore_conn_id=&apos;google_cloud_datastore_default&apos;, delegate_to=None)</pre>
<p>Bases: <a class="reference internal" href="#airflow.contrib.hooks.gcp_api_base_hook.GoogleCloudBaseHook" title="airflow.contrib.hooks.gcp_api_base_hook.GoogleCloudBaseHook"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.contrib.hooks.gcp_api_base_hook.GoogleCloudBaseHook</span></code></a></p>
<p>Interact with Google Cloud Datastore. This hook uses the Google Cloud Platform
connection.</p>
<p>This object is not threads safe. If you want to make multiple requests
simultaneously, you will need to create a hook per thread.</p>

<pre>
allocate_ids(partialKeys)</pre>
<p>Allocate IDs for incomplete keys.
see <a class="reference external" href="https://cloud.google.com/datastore/docs/reference/rest/v1/projects/allocateIds">https://cloud.google.com/datastore/docs/reference/rest/v1/projects/allocateIds</a></p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><strong>partialKeys</strong> &#x2013; a list of partial keys</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th>
<td class="field-body">a list of full keys.</td>
</tr>
</tbody>
</table>




<pre>
begin_transaction()</pre>
<p>Get a new transaction handle</p>
<blockquote>
<div><div class="admonition seealso">
<p class="first admonition-title">See also</p>
<p class="last"><a class="reference external" href="https://cloud.google.com/datastore/docs/reference/rest/v1/projects/beginTransaction">https://cloud.google.com/datastore/docs/reference/rest/v1/projects/beginTransaction</a></p>
</div>
</div>
</blockquote>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Returns:</th>
<td class="field-body">a transaction handle</td>
</tr>
</tbody>
</table>




<pre>
commit(body)</pre>
<p>Commit a transaction, optionally creating, deleting or modifying some entities.</p>
<div class="admonition seealso">
<p class="first admonition-title">See also</p>
<p class="last"><a class="reference external" href="https://cloud.google.com/datastore/docs/reference/rest/v1/projects/commit">https://cloud.google.com/datastore/docs/reference/rest/v1/projects/commit</a></p>
</div>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><strong>body</strong> &#x2013; the body of the commit request</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th>
<td class="field-body">the response body of the commit request</td>
</tr>
</tbody>
</table>




<pre>
delete_operation(name)</pre>
<p>Deletes the long-running operation</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><strong>name</strong> &#x2013; the name of the operation resource</td>
</tr>
</tbody>
</table>




<pre>
export_to_storage_bucket(bucket, namespace=None, entity_filter=None, labels=None)</pre>
<p>Export entities from Cloud Datastore to Cloud Storage for backup</p>




<pre>
get_conn(version=&apos;v1&apos;)</pre>
<p>Returns a Google Cloud Storage service object.</p>




<pre>
get_operation(name)</pre>
<p>Gets the latest state of a long-running operation</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><strong>name</strong> &#x2013; the name of the operation resource</td>
</tr>
</tbody>
</table>




<pre>
import_from_storage_bucket(bucket, file, namespace=None, entity_filter=None, labels=None)</pre>
<p>Import a backup from Cloud Storage to Cloud Datastore</p>




<pre>
lookup(keys, read_consistency=None, transaction=None)</pre>
<p>Lookup some entities by key</p>
<div class="admonition seealso">
<p class="first admonition-title">See also</p>
<p class="last"><a class="reference external" href="https://cloud.google.com/datastore/docs/reference/rest/v1/projects/lookup">https://cloud.google.com/datastore/docs/reference/rest/v1/projects/lookup</a></p>
</div>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first simple">
<li><strong>keys</strong> &#x2013; the keys to lookup</li>
<li><strong>read_consistency</strong> &#x2013; the read consistency to use. default, strong or eventual.
Cannot be used with a transaction.</li>
<li><strong>transaction</strong> &#x2013; the transaction to use, if any.</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th>
<td class="field-body"><p class="first last">the response body of the lookup request.</p>
</td>
</tr>
</tbody>
</table>




<pre>
poll_operation_until_done(name, polling_interval_in_seconds)</pre>
<p>Poll backup operation state until it&#x2019;s completed</p>




<pre>
rollback(transaction)</pre>
<p>Roll back a transaction</p>
<div class="admonition seealso">
<p class="first admonition-title">See also</p>
<p class="last"><a class="reference external" href="https://cloud.google.com/datastore/docs/reference/rest/v1/projects/rollback">https://cloud.google.com/datastore/docs/reference/rest/v1/projects/rollback</a></p>
</div>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><strong>transaction</strong> &#x2013; the transaction to roll back</td>
</tr>
</tbody>
</table>




<pre>
run_query(body)</pre>
<p>Run a query for entities.</p>
<div class="admonition seealso">
<p class="first admonition-title">See also</p>
<p class="last"><a class="reference external" href="https://cloud.google.com/datastore/docs/reference/rest/v1/projects/runQuery">https://cloud.google.com/datastore/docs/reference/rest/v1/projects/runQuery</a></p>
</div>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><strong>body</strong> &#x2013; the body of the query request</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th>
<td class="field-body">the batch of query results.</td>
</tr>
</tbody>
</table>







<pre>
class airflow.contrib.hooks.discord_webhook_hook.DiscordWebhookHook(http_conn_id=None, webhook_endpoint=None, message=&apos;&apos;, username=None, avatar_url=None, tts=False, proxy=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.hooks.http_hook.HttpHook" title="airflow.hooks.http_hook.HttpHook"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.hooks.http_hook.HttpHook</span></code></a></p>
<p>This hook allows you to post messages to Discord using incoming webhooks.
Takes a Discord connection ID with a default relative webhook endpoint. The
default endpoint can be overridden using the webhook_endpoint parameter
(<a class="reference external" href="https://discordapp.com/developers/docs/resources/webhook">https://discordapp.com/developers/docs/resources/webhook</a>).</p>
<p>Each Discord webhook can be pre-configured to use a specific username and
avatar_url. You can override these defaults in this hook.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>http_conn_id</strong> (<em>str</em>) &#x2013; Http connection ID with host as &#x201C;<a class="reference external" href="https://discord.com/api/">https://discord.com/api/</a>&#x201D; and
default webhook endpoint in the extra field in the form of
{&#x201C;webhook_endpoint&#x201D;: &#x201C;webhooks/{webhook.id}/{webhook.token}&#x201D;}</li>
<li><strong>webhook_endpoint</strong> (<em>str</em>) &#x2013; Discord webhook endpoint in the form of
&#x201C;webhooks/{webhook.id}/{webhook.token}&#x201D;</li>
<li><strong>message</strong> (<em>str</em>) &#x2013; The message you want to send to your Discord channel
(max 2000 characters)</li>
<li><strong>username</strong> (<em>str</em>) &#x2013; Override the default username of the webhook</li>
<li><strong>avatar_url</strong> (<em>str</em>) &#x2013; Override the default avatar of the webhook</li>
<li><strong>tts</strong> (<em>bool</em>) &#x2013; Is a text-to-speech message</li>
<li><strong>proxy</strong> (<em>str</em>) &#x2013; Proxy to use to make the Discord webhook call</li>
</ul>
</td>
</tr>
</tbody>
</table>

<pre>
execute()</pre>
<p>Execute the Discord webhook call</p>







<pre>
class airflow.contrib.hooks.emr_hook.EmrHook(emr_conn_id=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.contrib.hooks.aws_hook.AwsHook" title="airflow.contrib.hooks.aws_hook.AwsHook"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.contrib.hooks.aws_hook.AwsHook</span></code></a></p>
<p>Interact with AWS EMR. emr_conn_id is only neccessary for using the
create_job_flow method.</p>

<pre>
create_job_flow(job_flow_overrides)</pre>
<p>Creates a job flow using the config from the EMR connection.
Keys of the json extra hash may have the arguments of the boto3
run_job_flow method.
Overrides for this config may be passed as the job_flow_overrides.</p>







<pre>
class airflow.contrib.hooks.fs_hook.FSHook(conn_id=&apos;fs_default&apos;)</pre>
<p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.hooks.base_hook.BaseHook</span></code></p>
<p>Allows for interaction with an file server.</p>
<p>Connection should have a name and a path specified under extra:</p>
<p>example:
Conn Id: fs_test
Conn Type: File (path)
Host, Shchema, Login, Password, Port: empty
Extra: {&#x201C;path&#x201D;: &#x201C;/tmp&#x201D;}</p>




<pre>
class airflow.contrib.hooks.ftp_hook.FTPHook(ftp_conn_id=&apos;ftp_default&apos;)</pre>
<p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.hooks.base_hook.BaseHook</span></code>, <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.utils.log.logging_mixin.LoggingMixin</span></code></p>
<p>Interact with FTP.</p>
<p>Errors that may occur throughout but should be handled
downstream.</p>

<pre>
close_conn()</pre>
<p>Closes the connection. An error will occur if the
connection wasn&#x2019;t ever opened.</p>




<pre>
create_directory(path)</pre>
<p>Creates a directory on the remote system.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><strong>path</strong> (<em>str</em>) &#x2013; full path to the remote directory to create</td>
</tr>
</tbody>
</table>




<pre>
delete_directory(path)</pre>
<p>Deletes a directory on the remote system.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><strong>path</strong> (<em>str</em>) &#x2013; full path to the remote directory to delete</td>
</tr>
</tbody>
</table>




<pre>
delete_file(path)</pre>
<p>Removes a file on the FTP Server.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><strong>path</strong> (<em>str</em>) &#x2013; full path to the remote file</td>
</tr>
</tbody>
</table>




<pre>
describe_directory(path)</pre>
<p>Returns a dictionary of {filename: {attributes}} for all files
on the remote system (where the MLSD command is supported).</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><strong>path</strong> (<em>str</em>) &#x2013; full path to the remote directory</td>
</tr>
</tbody>
</table>




<pre>
get_conn()</pre>
<p>Returns a FTP connection object</p>




<pre>
list_directory(path, nlst=False)</pre>
<p>Returns a list of files on the remote system.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><strong>path</strong> (<em>str</em>) &#x2013; full path to the remote directory to list</td>
</tr>
</tbody>
</table>




<pre>
rename(from_name, to_name)</pre>
<p>Rename a file.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>from_name</strong> &#x2013; rename file from name</li>
<li><strong>to_name</strong> &#x2013; rename file to name</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
retrieve_file(remote_full_path, local_full_path_or_buffer)</pre>
<p>Transfers the remote file to a local location.</p>
<p>If local_full_path_or_buffer is a string path, the file will be put
at that location; if it is a file-like buffer, the file will
be written to the buffer but not closed.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>remote_full_path</strong> (<em>str</em>) &#x2013; full path to the remote file</li>
<li><strong>local_full_path_or_buffer</strong> (<em>str</em><em> or </em><em>file-like buffer</em>) &#x2013; full path to the local file or a
file-like buffer</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
store_file(remote_full_path, local_full_path_or_buffer)</pre>
<p>Transfers a local file to the remote location.</p>
<p>If local_full_path_or_buffer is a string path, the file will be read
from that location; if it is a file-like buffer, the file will
be read from the buffer but not closed.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>remote_full_path</strong> (<em>str</em>) &#x2013; full path to the remote file</li>
<li><strong>local_full_path_or_buffer</strong> (<em>str</em><em> or </em><em>file-like buffer</em>) &#x2013; full path to the local file or a
file-like buffer</li>
</ul>
</td>
</tr>
</tbody>
</table>







<pre>
class airflow.contrib.hooks.ftp_hook.FTPSHook(ftp_conn_id=&apos;ftp_default&apos;)</pre>
<p>Bases: <a class="reference internal" href="#airflow.contrib.hooks.ftp_hook.FTPHook" title="airflow.contrib.hooks.ftp_hook.FTPHook"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.contrib.hooks.ftp_hook.FTPHook</span></code></a></p>

<pre>
get_conn()</pre>
<p>Returns a FTPS connection object.</p>







<pre>
class airflow.contrib.hooks.gcp_api_base_hook.GoogleCloudBaseHook(gcp_conn_id=&apos;google_cloud_default&apos;, delegate_to=None)</pre>
<p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.hooks.base_hook.BaseHook</span></code>, <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.utils.log.logging_mixin.LoggingMixin</span></code></p>
<p>A base hook for Google cloud-related hooks. Google cloud has a shared REST
API client that is built in the same way no matter which service you use.
This class helps construct and authorize the credentials needed to then
call apiclient.discovery.build() to actually discover and build a client
for a Google cloud service.</p>
<p>The class also contains some miscellaneous helper functions.</p>
<p>All hook derived from this base hook use the &#x2018;Google Cloud Platform&#x2019; connection
type. Two ways of authentication are supported:</p>
<p>Default credentials: Only the &#x2018;Project Id&#x2019; is required. You&#x2019;ll need to
have set up default credentials, such as by the
<code class="docutils literal notranslate"><span class="pre">GOOGLE_APPLICATION_DEFAULT</span></code> environment variable or from the metadata
server on Google Compute Engine.</p>
<p>JSON key file: Specify &#x2018;Project Id&#x2019;, &#x2018;Key Path&#x2019; and &#x2018;Scope&#x2019;.</p>
<p>Legacy P12 key files are not supported.</p>




<pre>
class airflow.contrib.hooks.gcp_container_hook.GKEClusterHook(project_id, location)</pre>
<p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.hooks.base_hook.BaseHook</span></code></p>

<pre>
create_cluster(cluster, retry=&lt;object object&gt;, timeout=&lt;object object&gt;)</pre>
<p>Creates a cluster, consisting of the specified number and type of Google Compute
Engine instances.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first simple">
<li><strong>cluster</strong> (<em>dict</em><em> or </em><em>google.cloud.container_v1.types.Cluster</em>) &#x2013; A Cluster protobuf or dict. If dict is provided, it must be of
the same form as the protobuf message google.cloud.container_v1.types.Cluster</li>
<li><strong>retry</strong> (<em>google.api_core.retry.Retry</em>) &#x2013; A retry object (google.api_core.retry.Retry) used to retry requests.
If None is specified, requests will not be retried.</li>
<li><strong>timeout</strong> (<em>float</em>) &#x2013; The amount of time, in seconds, to wait for the request to
complete. Note that if retry is specified, the timeout applies to each
individual attempt.</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th>
<td class="field-body"><p class="first last">The full url to the new, or existing, cluster</p>
</td>
</tr>
</tbody>
</table>

<pre>:raises</pre>
ParseError: On JSON parsing problems when trying to convert dict
AirflowException: cluster is not dict type nor Cluster proto type





<pre>
delete_cluster(name, retry=&lt;object object&gt;, timeout=&lt;object object&gt;)</pre>
<p>Deletes the cluster, including the Kubernetes endpoint and all
worker nodes. Firewalls and routes that were configured during
cluster creation are also deleted. Other Google Compute Engine
resources that might be in use by the cluster (e.g. load balancer
resources) will not be deleted if they weren&#x2019;t present at the
initial create time.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first simple">
<li><strong>name</strong> (<em>str</em>) &#x2013; The name of the cluster to delete</li>
<li><strong>retry</strong> (<em>google.api_core.retry.Retry</em>) &#x2013; Retry object used to determine when/if to retry requests.
If None is specified, requests will not be retried.</li>
<li><strong>timeout</strong> (<em>float</em>) &#x2013; The amount of time, in seconds, to wait for the request to
complete. Note that if retry is specified, the timeout applies to each
individual attempt.</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th>
<td class="field-body"><p class="first last">The full url to the delete operation if successful, else None</p>
</td>
</tr>
</tbody>
</table>




<pre>
get_cluster(name, retry=&lt;object object&gt;, timeout=&lt;object object&gt;)</pre>
<p>Gets details of specified cluster
:param name: The name of the cluster to retrieve
:type name: str
:param retry: A retry object used to retry requests. If None is specified,</p>
<blockquote>
<div>requests will not be retried.</div>
</blockquote>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><strong>timeout</strong> (<em>float</em>) &#x2013; The amount of time, in seconds, to wait for the request to
complete. Note that if retry is specified, the timeout applies to each
individual attempt.</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th>
<td class="field-body">A google.cloud.container_v1.types.Cluster instance</td>
</tr>
</tbody>
</table>




<pre>
get_operation(operation_name)</pre>
<p>Fetches the operation from Google Cloud
:param operation_name: Name of operation to fetch
:type operation_name: str
:return: The new, updated operation from Google Cloud</p>




<pre>
wait_for_operation(operation)</pre>
<p>Given an operation, continuously fetches the status from Google Cloud until either
completion or an error occurring
:param operation: The Operation to wait for
:type operation: A google.cloud.container_V1.gapic.enums.Operator
:return: A new, updated operation fetched from Google Cloud</p>







<pre>
class airflow.contrib.hooks.gcp_dataflow_hook.DataFlowHook(gcp_conn_id=&apos;google_cloud_default&apos;, delegate_to=None, poll_sleep=10)</pre>
<p>Bases: <a class="reference internal" href="#airflow.contrib.hooks.gcp_api_base_hook.GoogleCloudBaseHook" title="airflow.contrib.hooks.gcp_api_base_hook.GoogleCloudBaseHook"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.contrib.hooks.gcp_api_base_hook.GoogleCloudBaseHook</span></code></a></p>

<pre>
get_conn()</pre>
<p>Returns a Google Cloud Storage service object.</p>







<pre>
class airflow.contrib.hooks.gcp_dataproc_hook.DataProcHook(gcp_conn_id=&apos;google_cloud_default&apos;, delegate_to=None, api_version=&apos;v1beta2&apos;)</pre>
<p>Bases: <a class="reference internal" href="#airflow.contrib.hooks.gcp_api_base_hook.GoogleCloudBaseHook" title="airflow.contrib.hooks.gcp_api_base_hook.GoogleCloudBaseHook"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.contrib.hooks.gcp_api_base_hook.GoogleCloudBaseHook</span></code></a></p>
<p>Hook for Google Cloud Dataproc APIs.</p>

<pre>
await(operation)</pre>
<p>Awaits for Google Cloud Dataproc Operation to complete.</p>




<pre>
get_conn()</pre>
<p>Returns a Google Cloud Dataproc service object.</p>




<pre>
wait(operation)</pre>
<p>Awaits for Google Cloud Dataproc Operation to complete.</p>







<pre>
class airflow.contrib.hooks.gcp_mlengine_hook.MLEngineHook(gcp_conn_id=&apos;google_cloud_default&apos;, delegate_to=None)</pre>
<p>Bases: <a class="reference internal" href="#airflow.contrib.hooks.gcp_api_base_hook.GoogleCloudBaseHook" title="airflow.contrib.hooks.gcp_api_base_hook.GoogleCloudBaseHook"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.contrib.hooks.gcp_api_base_hook.GoogleCloudBaseHook</span></code></a></p>

<pre>
create_job(project_id, job, use_existing_job_fn=None)</pre>
<p>Launches a MLEngine job and wait for it to reach a terminal state.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first simple">
<li><strong>project_id</strong> (<em>string</em>) &#x2013; The Google Cloud project id within which MLEngine
job will be launched.</li>
<li><strong>job</strong> (<em>dict</em>) &#x2013; <p>MLEngine Job object that should be provided to the MLEngine
API, such as:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="p">{</span>
  <span class="s1">&apos;jobId&apos;</span><span class="p">:</span> <span class="s1">&apos;my_job_id&apos;</span><span class="p">,</span>
  <span class="s1">&apos;trainingInput&apos;</span><span class="p">:</span> <span class="p">{</span>
    <span class="s1">&apos;scaleTier&apos;</span><span class="p">:</span> <span class="s1">&apos;STANDARD_1&apos;</span><span class="p">,</span>
    <span class="o">...</span>
  <span class="p">}</span>
<span class="p">}</span>
</pre>
</div>
</div>
</li>
<li><strong>use_existing_job_fn</strong> (<em>function</em>) &#x2013; In case that a MLEngine job with the same
job_id already exist, this method (if provided) will decide whether
we should use this existing job, continue waiting for it to finish
and returning the job object. It should accepts a MLEngine job
object, and returns a boolean value indicating whether it is OK to
reuse the existing job. If &#x2018;use_existing_job_fn&#x2019; is not provided,
we by default reuse the existing MLEngine job.</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th>
<td class="field-body"><p class="first">The MLEngine job object if the job successfully reach a
terminal state (which might be FAILED or CANCELLED state).</p>
</td>
</tr>
<tr class="field-odd field"><th class="field-name">Return type:</th>
<td class="field-body"><p class="first last">dict</p>
</td>
</tr>
</tbody>
</table>




<pre>
create_model(project_id, model)</pre>
<p>Create a Model. Blocks until finished.</p>




<pre>
create_version(project_id, model_name, version_spec)</pre>
<p>Creates the Version on Google Cloud ML Engine.</p>
<p>Returns the operation if the version was created successfully and
raises an error otherwise.</p>




<pre>
delete_version(project_id, model_name, version_name)</pre>
<p>Deletes the given version of a model. Blocks until finished.</p>




<pre>
get_conn()</pre>
<p>Returns a Google MLEngine service object.</p>




<pre>
get_model(project_id, model_name)</pre>
<p>Gets a Model. Blocks until finished.</p>




<pre>
list_versions(project_id, model_name)</pre>
<p>Lists all available versions of a model. Blocks until finished.</p>




<pre>
set_default_version(project_id, model_name, version_name)</pre>
<p>Sets a version to be the default. Blocks until finished.</p>







<pre>
class airflow.contrib.hooks.gcp_pubsub_hook.PubSubHook(gcp_conn_id=&apos;google_cloud_default&apos;, delegate_to=None)</pre>
<p>Bases: <a class="reference internal" href="#airflow.contrib.hooks.gcp_api_base_hook.GoogleCloudBaseHook" title="airflow.contrib.hooks.gcp_api_base_hook.GoogleCloudBaseHook"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.contrib.hooks.gcp_api_base_hook.GoogleCloudBaseHook</span></code></a></p>
<p>Hook for accessing Google Pub/Sub.</p>
<p>The GCP project against which actions are applied is determined by
the project embedded in the Connection referenced by gcp_conn_id.</p>

<pre>
acknowledge(project, subscription, ack_ids)</pre>
<p>Pulls up to <code class="docutils literal notranslate"><span class="pre">max_messages</span></code> messages from Pub/Sub subscription.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>project</strong> (<em>string</em>) &#x2013; the GCP project name or ID in which to create
the topic</li>
<li><strong>subscription</strong> (<em>string</em>) &#x2013; the Pub/Sub subscription name to delete; do not
include the &#x2018;projects/{project}/topics/&#x2019; prefix.</li>
<li><strong>ack_ids</strong> (<em>list</em>) &#x2013; List of ReceivedMessage ackIds from a previous pull
response</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
create_subscription(topic_project, topic, subscription=None, subscription_project=None, ack_deadline_secs=10, fail_if_exists=False)</pre>
<p>Creates a Pub/Sub subscription, if it does not already exist.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first simple">
<li><strong>topic_project</strong> (<em>string</em>) &#x2013; the GCP project ID of the topic that the
subscription will be bound to.</li>
<li><strong>topic</strong> (<em>string</em>) &#x2013; the Pub/Sub topic name that the subscription will be bound
to create; do not include the <code class="docutils literal notranslate"><span class="pre">projects/{project}/subscriptions/</span></code>
prefix.</li>
<li><strong>subscription</strong> (<em>string</em>) &#x2013; the Pub/Sub subscription name. If empty, a random
name will be generated using the uuid module</li>
<li><strong>subscription_project</strong> (<em>string</em>) &#x2013; the GCP project ID where the subscription
will be created. If unspecified, <code class="docutils literal notranslate"><span class="pre">topic_project</span></code> will be used.</li>
<li><strong>ack_deadline_secs</strong> (<em>int</em>) &#x2013; Number of seconds that a subscriber has to
acknowledge each message pulled from the subscription</li>
<li><strong>fail_if_exists</strong> (<em>bool</em>) &#x2013; if set, raise an exception if the topic
already exists</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th>
<td class="field-body"><p class="first">subscription name which will be the system-generated value if
the <code class="docutils literal notranslate"><span class="pre">subscription</span></code> parameter is not supplied</p>
</td>
</tr>
<tr class="field-odd field"><th class="field-name">Return type:</th>
<td class="field-body"><p class="first last">string</p>
</td>
</tr>
</tbody>
</table>




<pre>
create_topic(project, topic, fail_if_exists=False)</pre>
<p>Creates a Pub/Sub topic, if it does not already exist.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>project</strong> (<em>string</em>) &#x2013; the GCP project ID in which to create
the topic</li>
<li><strong>topic</strong> (<em>string</em>) &#x2013; the Pub/Sub topic name to create; do not
include the <code class="docutils literal notranslate"><span class="pre">projects/{project}/topics/</span></code> prefix.</li>
<li><strong>fail_if_exists</strong> (<em>bool</em>) &#x2013; if set, raise an exception if the topic
already exists</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
delete_subscription(project, subscription, fail_if_not_exists=False)</pre>
<p>Deletes a Pub/Sub subscription, if it exists.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>project</strong> (<em>string</em>) &#x2013; the GCP project ID where the subscription exists</li>
<li><strong>subscription</strong> (<em>string</em>) &#x2013; the Pub/Sub subscription name to delete; do not
include the <code class="docutils literal notranslate"><span class="pre">projects/{project}/subscriptions/</span></code> prefix.</li>
<li><strong>fail_if_not_exists</strong> (<em>bool</em>) &#x2013; if set, raise an exception if the topic
does not exist</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
delete_topic(project, topic, fail_if_not_exists=False)</pre>
<p>Deletes a Pub/Sub topic if it exists.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>project</strong> (<em>string</em>) &#x2013; the GCP project ID in which to delete the topic</li>
<li><strong>topic</strong> (<em>string</em>) &#x2013; the Pub/Sub topic name to delete; do not
include the <code class="docutils literal notranslate"><span class="pre">projects/{project}/topics/</span></code> prefix.</li>
<li><strong>fail_if_not_exists</strong> (<em>bool</em>) &#x2013; if set, raise an exception if the topic
does not exist</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
get_conn()</pre>
<p>Returns a Pub/Sub service object.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Return type:</th>
<td class="field-body">apiclient.discovery.Resource</td>
</tr>
</tbody>
</table>




<pre>
publish(project, topic, messages)</pre>
<p>Publishes messages to a Pub/Sub topic.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>project</strong> (<em>string</em>) &#x2013; the GCP project ID in which to publish</li>
<li><strong>topic</strong> (<em>string</em>) &#x2013; the Pub/Sub topic to which to publish; do not
include the <code class="docutils literal notranslate"><span class="pre">projects/{project}/topics/</span></code> prefix.</li>
<li><strong>messages</strong> (list of PubSub messages; see
<a class="reference external" href="http://cloud.google.com/pubsub/docs/reference/rest/v1/PubsubMessage">http://cloud.google.com/pubsub/docs/reference/rest/v1/PubsubMessage</a>) &#x2013; messages to publish; if the data field in a
message is set, it should already be base64 encoded.</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
pull(project, subscription, max_messages, return_immediately=False)</pre>
<p>Pulls up to <code class="docutils literal notranslate"><span class="pre">max_messages</span></code> messages from Pub/Sub subscription.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>project</strong> (<em>string</em>) &#x2013; the GCP project ID where the subscription exists</li>
<li><strong>subscription</strong> (<em>string</em>) &#x2013; the Pub/Sub subscription name to pull from; do not
include the &#x2018;projects/{project}/topics/&#x2019; prefix.</li>
<li><strong>max_messages</strong> (<em>int</em>) &#x2013; The maximum number of messages to return from
the Pub/Sub API.</li>
<li><strong>return_immediately</strong> (<em>bool</em>) &#x2013; If set, the Pub/Sub API will immediately
return if no messages are available. Otherwise, the request will
block for an undisclosed, but bounded period of time</li>
</ul>
</td>
</tr>
</tbody>
</table>

<pre>:return A list of Pub/Sub ReceivedMessage objects each containing</pre>
an <code class="docutils literal notranslate"><span class="pre">ackId</span></code> property and a <code class="docutils literal notranslate"><span class="pre">message</span></code> property, which includes
the base64-encoded message content. See
<a class="reference external" href="https://cloud.google.com/pubsub/docs/reference/rest/v1/">https://cloud.google.com/pubsub/docs/reference/rest/v1/</a>                projects.subscriptions/pull#ReceivedMessage








<pre>
class airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook(google_cloud_storage_conn_id=&apos;google_cloud_default&apos;, delegate_to=None)</pre>
<p>Bases: <a class="reference internal" href="#airflow.contrib.hooks.gcp_api_base_hook.GoogleCloudBaseHook" title="airflow.contrib.hooks.gcp_api_base_hook.GoogleCloudBaseHook"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.contrib.hooks.gcp_api_base_hook.GoogleCloudBaseHook</span></code></a></p>
<p>Interact with Google Cloud Storage. This hook uses the Google Cloud Platform
connection.</p>

<pre>
copy(source_bucket, source_object, destination_bucket=None, destination_object=None)</pre>
<p>Copies an object from a bucket to another, with renaming if requested.</p>
<p>destination_bucket or destination_object can be omitted, in which case
source bucket/object is used, but not both.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>source_bucket</strong> (<em>string</em>) &#x2013; The bucket of the object to copy from.</li>
<li><strong>source_object</strong> (<em>string</em>) &#x2013; The object to copy.</li>
<li><strong>destination_bucket</strong> (<em>string</em>) &#x2013; The destination of the object to copied to.
Can be omitted; then the same bucket is used.</li>
<li><strong>destination_object</strong> &#x2013; The (renamed) path of the object if given.
Can be omitted; then the same name is used.</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
create_bucket(bucket_name, storage_class=&apos;MULTI_REGIONAL&apos;, location=&apos;US&apos;, project_id=None, labels=None)</pre>
<p>Creates a new bucket. Google Cloud Storage uses a flat namespace, so
you can&#x2019;t create a bucket with a name that is already in use.</p>
<div class="admonition seealso">
<p class="first admonition-title">See also</p>
<p class="last">For more information, see Bucket Naming Guidelines:
<a class="reference external" href="https://cloud.google.com/storage/docs/bucketnaming.html#requirements">https://cloud.google.com/storage/docs/bucketnaming.html#requirements</a></p>
</div>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first simple">
<li><strong>bucket_name</strong> (<em>string</em>) &#x2013; The name of the bucket.</li>
<li><strong>storage_class</strong> (<em>string</em>) &#x2013; <p>This defines how objects in the bucket are stored
and determines the SLA and the cost of storage. Values include</p>
<ul>
<li><code class="docutils literal notranslate"><span class="pre">MULTI_REGIONAL</span></code></li>
<li><code class="docutils literal notranslate"><span class="pre">REGIONAL</span></code></li>
<li><code class="docutils literal notranslate"><span class="pre">STANDARD</span></code></li>
<li><code class="docutils literal notranslate"><span class="pre">NEARLINE</span></code></li>
<li><code class="docutils literal notranslate"><span class="pre">COLDLINE</span></code>.</li>
</ul>
<p>If this value is not specified when the bucket is
created, it will default to STANDARD.</p>
</li>
<li><strong>location</strong> (<em>string</em>) &#x2013; <p>The location of the bucket.
Object data for objects in the bucket resides in physical storage
within this region. Defaults to US.</p>
<div class="admonition seealso">
<p class="first admonition-title">See also</p>
<p class="last"><a class="reference external" href="https://developers.google.com/storage/docs/bucket-locations">https://developers.google.com/storage/docs/bucket-locations</a></p>
</div>
</li>
<li><strong>project_id</strong> (<em>string</em>) &#x2013; The ID of the GCP Project.</li>
<li><strong>labels</strong> (<em>dict</em>) &#x2013; User-provided labels, in key/value pairs.</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th>
<td class="field-body"><p class="first last">If successful, it returns the <code class="docutils literal notranslate"><span class="pre">id</span></code> of the bucket.</p>
</td>
</tr>
</tbody>
</table>




<pre>
delete(bucket, object, generation=None)</pre>
<p>Delete an object if versioning is not enabled for the bucket, or if generation
parameter is used.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first simple">
<li><strong>bucket</strong> (<em>string</em>) &#x2013; name of the bucket, where the object resides</li>
<li><strong>object</strong> (<em>string</em>) &#x2013; name of the object to delete</li>
<li><strong>generation</strong> (<em>string</em>) &#x2013; if present, permanently delete the object of this generation</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th>
<td class="field-body"><p class="first last">True if succeeded</p>
</td>
</tr>
</tbody>
</table>




<pre>
download(bucket, object, filename=None)</pre>
<p>Get a file from Google Cloud Storage.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>bucket</strong> (<em>string</em>) &#x2013; The bucket to fetch from.</li>
<li><strong>object</strong> (<em>string</em>) &#x2013; The object to fetch.</li>
<li><strong>filename</strong> (<em>string</em>) &#x2013; If set, a local file path where the file should be written to.</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
exists(bucket, object)</pre>
<p>Checks for the existence of a file in Google Cloud Storage.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>bucket</strong> (<em>string</em>) &#x2013; The Google cloud storage bucket where the object is.</li>
<li><strong>object</strong> (<em>string</em>) &#x2013; The name of the object to check in the Google cloud
storage bucket.</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
get_conn()</pre>
<p>Returns a Google Cloud Storage service object.</p>




<pre>
get_crc32c(bucket, object)</pre>
<p>Gets the CRC32c checksum of an object in Google Cloud Storage.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>bucket</strong> (<em>string</em>) &#x2013; The Google cloud storage bucket where the object is.</li>
<li><strong>object</strong> (<em>string</em>) &#x2013; The name of the object to check in the Google cloud
storage bucket.</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
get_md5hash(bucket, object)</pre>
<p>Gets the MD5 hash of an object in Google Cloud Storage.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>bucket</strong> (<em>string</em>) &#x2013; The Google cloud storage bucket where the object is.</li>
<li><strong>object</strong> (<em>string</em>) &#x2013; The name of the object to check in the Google cloud
storage bucket.</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
get_size(bucket, object)</pre>
<p>Gets the size of a file in Google Cloud Storage.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>bucket</strong> (<em>string</em>) &#x2013; The Google cloud storage bucket where the object is.</li>
<li><strong>object</strong> (<em>string</em>) &#x2013; The name of the object to check in the Google cloud storage bucket.</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
is_updated_after(bucket, object, ts)</pre>
<p>Checks if an object is updated in Google Cloud Storage.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>bucket</strong> (<em>string</em>) &#x2013; The Google cloud storage bucket where the object is.</li>
<li><strong>object</strong> (<em>string</em>) &#x2013; The name of the object to check in the Google cloud
storage bucket.</li>
<li><strong>ts</strong> (<em>datetime</em>) &#x2013; The timestamp to check against.</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
list(bucket, versions=None, maxResults=None, prefix=None, delimiter=None)</pre>
<p>List all objects from the bucket with the give string prefix in name</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first simple">
<li><strong>bucket</strong> (<em>string</em>) &#x2013; bucket name</li>
<li><strong>versions</strong> (<em>boolean</em>) &#x2013; if true, list all versions of the objects</li>
<li><strong>maxResults</strong> (<em>integer</em>) &#x2013; max count of items to return in a single page of responses</li>
<li><strong>prefix</strong> (<em>string</em>) &#x2013; prefix string which filters objects whose name begin with
this prefix</li>
<li><strong>delimiter</strong> (<em>string</em>) &#x2013; filters objects based on the delimiter (for e.g &#x2018;.csv&#x2019;)</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th>
<td class="field-body"><p class="first last">a stream of object names matching the filtering criteria</p>
</td>
</tr>
</tbody>
</table>




<pre>
rewrite(source_bucket, source_object, destination_bucket, destination_object=None)</pre>
<p>Has the same functionality as copy, except that will work on files
over 5 TB, as well as when copying between locations and/or storage
classes.</p>
<p>destination_object can be omitted, in which case source_object is used.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>source_bucket</strong> (<em>string</em>) &#x2013; The bucket of the object to copy from.</li>
<li><strong>source_object</strong> (<em>string</em>) &#x2013; The object to copy.</li>
<li><strong>destination_bucket</strong> (<em>string</em>) &#x2013; The destination of the object to copied to.</li>
<li><strong>destination_object</strong> &#x2013; The (renamed) path of the object if given.
Can be omitted; then the same name is used.</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
upload(bucket, object, filename, mime_type=&apos;application/octet-stream&apos;)</pre>
<p>Uploads a local file to Google Cloud Storage.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>bucket</strong> (<em>string</em>) &#x2013; The bucket to upload to.</li>
<li><strong>object</strong> (<em>string</em>) &#x2013; The object name to set when uploading the local file.</li>
<li><strong>filename</strong> (<em>string</em>) &#x2013; The local file path to the file to be uploaded.</li>
<li><strong>mime_type</strong> (<em>string</em>) &#x2013; The MIME type to set when uploading the file.</li>
</ul>
</td>
</tr>
</tbody>
</table>







<pre>
class airflow.contrib.hooks.jenkins_hook.JenkinsHook(conn_id=&apos;jenkins_default&apos;)</pre>
<p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.hooks.base_hook.BaseHook</span></code></p>
<p>Hook to manage connection to jenkins server</p>




<pre>
class airflow.contrib.hooks.jira_hook.JiraHook(jira_conn_id=&apos;jira_default&apos;, proxies=None)</pre>
<p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.hooks.base_hook.BaseHook</span></code>, <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.utils.log.logging_mixin.LoggingMixin</span></code></p>
<p>Jira interaction hook, a Wrapper around JIRA Python SDK.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><strong>jira_conn_id</strong> (<em>string</em>) &#x2013; reference to a pre-defined Jira Connection</td>
</tr>
</tbody>
</table>




<pre>
class airflow.contrib.hooks.mongo_hook.MongoHook(conn_id=&apos;mongo_default&apos;, *args, **kwargs)</pre>
<p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.hooks.base_hook.BaseHook</span></code></p>
<p>PyMongo Wrapper to Interact With Mongo Database
Mongo Connection Documentation
<a class="reference external" href="https://docs.mongodb.com/manual/reference/connection-string/index.html">https://docs.mongodb.com/manual/reference/connection-string/index.html</a>
You can specify connection string options in extra field of your connection
<a class="reference external" href="https://docs.mongodb.com/manual/reference/connection-string/index.html#connection-string-options">https://docs.mongodb.com/manual/reference/connection-string/index.html#connection-string-options</a>
ex.</p>
<blockquote>
<div>{replicaSet: test, ssl: True, connectTimeoutMS: 30000}</div>
</blockquote>

<pre>
aggregate(mongo_collection, aggregate_query, mongo_db=None, **kwargs)</pre>
<p>Runs an aggregation pipeline and returns the results
<a class="reference external" href="https://api.mongodb.com/python/current/api/pymongo/collection.html#pymongo.collection.Collection.aggregate">https://api.mongodb.com/python/current/api/pymongo/collection.html#pymongo.collection.Collection.aggregate</a>
<a class="reference external" href="https://api.mongodb.com/python/current/examples/aggregation.html">https://api.mongodb.com/python/current/examples/aggregation.html</a></p>




<pre>
find(mongo_collection, query, find_one=False, mongo_db=None, **kwargs)</pre>
<p>Runs a mongo find query and returns the results
<a class="reference external" href="https://api.mongodb.com/python/current/api/pymongo/collection.html#pymongo.collection.Collection.find">https://api.mongodb.com/python/current/api/pymongo/collection.html#pymongo.collection.Collection.find</a></p>




<pre>
get_collection(mongo_collection, mongo_db=None)</pre>
<p>Fetches a mongo collection object for querying.</p>
<p>Uses connection schema as DB unless specified.</p>




<pre>
get_conn()</pre>
<p>Fetches PyMongo Client</p>




<pre>
insert_many(mongo_collection, docs, mongo_db=None, **kwargs)</pre>
<p>Inserts many docs into a mongo collection.
<a class="reference external" href="https://api.mongodb.com/python/current/api/pymongo/collection.html#pymongo.collection.Collection.insert_many">https://api.mongodb.com/python/current/api/pymongo/collection.html#pymongo.collection.Collection.insert_many</a></p>




<pre>
insert_one(mongo_collection, doc, mongo_db=None, **kwargs)</pre>
<p>Inserts a single document into a mongo collection
<a class="reference external" href="https://api.mongodb.com/python/current/api/pymongo/collection.html#pymongo.collection.Collection.insert_one">https://api.mongodb.com/python/current/api/pymongo/collection.html#pymongo.collection.Collection.insert_one</a></p>







<pre>
class airflow.contrib.hooks.pinot_hook.PinotDbApiHook(*args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.hooks.dbapi_hook.DbApiHook" title="airflow.hooks.dbapi_hook.DbApiHook"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.hooks.dbapi_hook.DbApiHook</span></code></a></p>
<p>Connect to pinot db(<a class="reference external" href="https://github.com/linkedin/pinot">https://github.com/linkedin/pinot</a>) to issue pql</p>

<pre>
get_conn()</pre>
<p>Establish a connection to pinot broker through pinot dbqpi.</p>




<pre>
get_first(sql)</pre>
<p>Executes the sql and returns the first resulting row.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><strong>sql</strong> (<em>str</em><em> or </em><em>list</em>) &#x2013; the sql statement to be executed (str) or a list of
sql statements to execute</td>
</tr>
</tbody>
</table>




<pre>
get_pandas_df(sql, parameters=None)</pre>
<p>Executes the sql and returns a pandas dataframe</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>sql</strong> (<em>str</em><em> or </em><em>list</em>) &#x2013; the sql statement to be executed (str) or a list of
sql statements to execute</li>
<li><strong>parameters</strong> (<em>mapping</em><em> or </em><em>iterable</em>) &#x2013; The parameters to render the SQL query with.</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
get_records(sql)</pre>
<p>Executes the sql and returns a set of records.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><strong>sql</strong> (<em>str</em>) &#x2013; the sql statement to be executed (str) or a list of
sql statements to execute</td>
</tr>
</tbody>
</table>




<pre>
get_uri()</pre>
<p>Get the connection uri for pinot broker.</p>
<p>e.g: <a class="reference external" href="http://localhost:9000/pql">http://localhost:9000/pql</a></p>




<pre>
insert_rows(table, rows, target_fields=None, commit_every=1000)</pre>
<p>A generic way to insert a set of tuples into a table,
a new transaction is created every commit_every rows</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>table</strong> (<em>str</em>) &#x2013; Name of the target table</li>
<li><strong>rows</strong> (<em>iterable of tuples</em>) &#x2013; The rows to insert into the table</li>
<li><strong>target_fields</strong> (<em>iterable of strings</em>) &#x2013; The names of the columns to fill in the table</li>
<li><strong>commit_every</strong> (<em>int</em>) &#x2013; The maximum number of rows to insert in one
transaction. Set to 0 to insert all rows in one transaction.</li>
<li><strong>replace</strong> (<em>bool</em>) &#x2013; Whether to replace instead of insert</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
set_autocommit(conn, autocommit)</pre>
<p>Sets the autocommit flag on the connection</p>







<pre>
class airflow.contrib.hooks.qubole_hook.QuboleHook(*args, **kwargs)</pre>
<p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.hooks.base_hook.BaseHook</span></code>, <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.utils.log.logging_mixin.LoggingMixin</span></code></p>

<pre>
get_jobs_id(ti)</pre>
<p>Get jobs associated with a Qubole commands
:param ti: Task Instance of the dag, used to determine the Quboles command id
:return: Job informations assoiciated with command</p>




<pre>
get_log(ti)</pre>
<p>Get Logs of a command from Qubole
:param ti: Task Instance of the dag, used to determine the Quboles command id
:return: command log as text</p>




<pre>
get_results(ti=None, fp=None, inline=True, delim=None, fetch=True)</pre>
<p>Get results (or just s3 locations) of a command from Qubole and save into a file
:param ti: Task Instance of the dag, used to determine the Quboles command id
:param fp: Optional file pointer, will create one and return if None passed
:param inline: True to download actual results, False to get s3 locations only
:param delim: Replaces the CTL-A chars with the given delim, defaults to &#x2018;,&#x2019;
:param fetch: when inline is True, get results directly from s3 (if large)
:return: file location containing actual results or s3 locations of results</p>




<pre>
kill(ti)</pre>
<p>Kill (cancel) a Qubole commmand
:param ti: Task Instance of the dag, used to determine the Quboles command id
:return: response from Qubole</p>







<pre>
class airflow.contrib.hooks.redis_hook.RedisHook(redis_conn_id=&apos;redis_default&apos;)</pre>
<p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.hooks.base_hook.BaseHook</span></code>, <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.utils.log.logging_mixin.LoggingMixin</span></code></p>
<p>Hook to interact with Redis database</p>

<pre>
get_conn()</pre>
<p>Returns a Redis connection.</p>




<pre>
key_exists(key)</pre>
<p>Checks if a key exists in Redis database</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><strong>key</strong> (<em>string</em>) &#x2013; The key to check the existence.</td>
</tr>
</tbody>
</table>







<pre>
class airflow.contrib.hooks.redshift_hook.RedshiftHook(aws_conn_id=&apos;aws_default&apos;)</pre>
<p>Bases: <a class="reference internal" href="#airflow.contrib.hooks.aws_hook.AwsHook" title="airflow.contrib.hooks.aws_hook.AwsHook"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.contrib.hooks.aws_hook.AwsHook</span></code></a></p>
<p>Interact with AWS Redshift, using the boto3 library</p>

<pre>
cluster_status(cluster_identifier)</pre>
<p>Return status of a cluster</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><strong>cluster_identifier</strong> (<em>str</em>) &#x2013; unique identifier of a cluster</td>
</tr>
</tbody>
</table>




<pre>
create_cluster_snapshot(snapshot_identifier, cluster_identifier)</pre>
<p>Creates a snapshot of a cluster</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>snapshot_identifier</strong> (<em>str</em>) &#x2013; unique identifier for a snapshot of a cluster</li>
<li><strong>cluster_identifier</strong> (<em>str</em>) &#x2013; unique identifier of a cluster</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
delete_cluster(cluster_identifier, skip_final_cluster_snapshot=True, final_cluster_snapshot_identifier=&apos;&apos;)</pre>
<p>Delete a cluster and optionally create a snapshot</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>cluster_identifier</strong> (<em>str</em>) &#x2013; unique identifier of a cluster</li>
<li><strong>skip_final_cluster_snapshot</strong> (<em>bool</em>) &#x2013; determines cluster snapshot creation</li>
<li><strong>final_cluster_snapshot_identifier</strong> (<em>str</em>) &#x2013; name of final cluster snapshot</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
describe_cluster_snapshots(cluster_identifier)</pre>
<p>Gets a list of snapshots for a cluster</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><strong>cluster_identifier</strong> (<em>str</em>) &#x2013; unique identifier of a cluster</td>
</tr>
</tbody>
</table>




<pre>
restore_from_cluster_snapshot(cluster_identifier, snapshot_identifier)</pre>
<p>Restores a cluster from its snapshot</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>cluster_identifier</strong> (<em>str</em>) &#x2013; unique identifier of a cluster</li>
<li><strong>snapshot_identifier</strong> (<em>str</em>) &#x2013; unique identifier for a snapshot of a cluster</li>
</ul>
</td>
</tr>
</tbody>
</table>







<pre>
class airflow.contrib.hooks.segment_hook.SegmentHook(segment_conn_id=&apos;segment_default&apos;, segment_debug_mode=False, *args, **kwargs)</pre>
<p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.hooks.base_hook.BaseHook</span></code>, <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.utils.log.logging_mixin.LoggingMixin</span></code></p>

<pre>
on_error(error, items)</pre>
<p>Handles error callbacks when using Segment with segment_debug_mode set to True</p>







<pre>
class airflow.contrib.hooks.sftp_hook.SFTPHook(ftp_conn_id=&apos;sftp_default&apos;)</pre>
<p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.hooks.base_hook.BaseHook</span></code></p>
<p>Interact with SFTP. Aims to be interchangeable with FTPHook.</p>

<pre>Pitfalls: - In contrast with FTPHook describe_directory only returns size, type and</pre>
<blockquote class="first">
<div>modify. It doesn&#x2019;t return unix.owner, unix.mode, perm, unix.group and
unique.</div>
</blockquote>
<ul class="last simple">
<li>retrieve_file and store_file only take a local full path and not a
buffer.</li>
<li>If no mode is passed to create_directory it will be created with 777
permissions.</li>
</ul>


<p>Errors that may occur throughout but should be handled downstream.</p>

<pre>
close_conn()</pre>
<p>Closes the connection. An error will occur if the
connection wasnt ever opened.</p>




<pre>
create_directory(path, mode=777)</pre>
<p>Creates a directory on the remote system.
:param path: full path to the remote directory to create
:type path: str
:param mode: int representation of octal mode for directory</p>




<pre>
delete_directory(path)</pre>
<p>Deletes a directory on the remote system.
:param path: full path to the remote directory to delete
:type path: str</p>




<pre>
delete_file(path)</pre>
<p>Removes a file on the FTP Server
:param path: full path to the remote file
:type path: str</p>




<pre>
describe_directory(path)</pre>
<p>Returns a dictionary of {filename: {attributes}} for all files
on the remote system (where the MLSD command is supported).
:param path: full path to the remote directory
:type path: str</p>




<pre>
get_conn()</pre>
<p>Returns an SFTP connection object</p>




<pre>
list_directory(path)</pre>
<p>Returns a list of files on the remote system.
:param path: full path to the remote directory to list
:type path: str</p>




<pre>
retrieve_file(remote_full_path, local_full_path)</pre>
<p>Transfers the remote file to a local location.
If local_full_path is a string path, the file will be put
at that location
:param remote_full_path: full path to the remote file
:type remote_full_path: str
:param local_full_path: full path to the local file
:type local_full_path: str</p>




<pre>
store_file(remote_full_path, local_full_path)</pre>
<p>Transfers a local file to the remote location.
If local_full_path_or_buffer is a string path, the file will be read
from that location
:param remote_full_path: full path to the remote file
:type remote_full_path: str
:param local_full_path: full path to the local file
:type local_full_path: str</p>







<pre>
class airflow.contrib.hooks.slack_webhook_hook.SlackWebhookHook(http_conn_id=None, webhook_token=None, message=&apos;&apos;, channel=None, username=None, icon_emoji=None, link_names=False, proxy=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.hooks.http_hook.HttpHook" title="airflow.hooks.http_hook.HttpHook"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.hooks.http_hook.HttpHook</span></code></a></p>
<p>This hook allows you to post messages to Slack using incoming webhooks.
Takes both Slack webhook token directly and connection that has Slack webhook token.
If both supplied, Slack webhook token will be used.</p>
<p>Each Slack webhook token can be pre-configured to use a specific channel, username and
icon. You can override these defaults in this hook.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>http_conn_id</strong> (<em>str</em>) &#x2013; connection that has Slack webhook token in the extra field</li>
<li><strong>webhook_token</strong> (<em>str</em>) &#x2013; Slack webhook token</li>
<li><strong>message</strong> (<em>str</em>) &#x2013; The message you want to send on Slack</li>
<li><strong>channel</strong> (<em>str</em>) &#x2013; The channel the message should be posted to</li>
<li><strong>username</strong> (<em>str</em>) &#x2013; The username to post to slack with</li>
<li><strong>icon_emoji</strong> (<em>str</em>) &#x2013; The emoji to use as icon for the user posting to Slack</li>
<li><strong>link_names</strong> (<em>bool</em>) &#x2013; Whether or not to find and link channel and usernames in your
message</li>
<li><strong>proxy</strong> (<em>str</em>) &#x2013; Proxy to use to make the Slack webhook call</li>
</ul>
</td>
</tr>
</tbody>
</table>

<pre>
execute()</pre>
<p>Remote Popen (actually execute the slack webhook call)</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>cmd</strong> &#x2013; command to remotely execute</li>
<li><strong>kwargs</strong> &#x2013; extra arguments to Popen (see subprocess.Popen)</li>
</ul>
</td>
</tr>
</tbody>
</table>







<pre>
class airflow.contrib.hooks.snowflake_hook.SnowflakeHook(*args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.hooks.dbapi_hook.DbApiHook" title="airflow.hooks.dbapi_hook.DbApiHook"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.hooks.dbapi_hook.DbApiHook</span></code></a></p>
<p>Interact with Snowflake.</p>
<p>get_sqlalchemy_engine() depends on snowflake-sqlalchemy</p>

<pre>
get_conn()</pre>
<p>Returns a snowflake.connection object</p>




<pre>
get_uri()</pre>
<p>override DbApiHook get_uri method for get_sqlalchemy_engine()</p>




<pre>
set_autocommit(conn, autocommit)</pre>
<p>Sets the autocommit flag on the connection</p>







<pre>
class airflow.contrib.hooks.spark_jdbc_hook.SparkJDBCHook(spark_app_name=&apos;airflow-spark-jdbc&apos;, spark_conn_id=&apos;spark-default&apos;, spark_conf=None, spark_py_files=None, spark_files=None, spark_jars=None, num_executors=None, executor_cores=None, executor_memory=None, driver_memory=None, verbose=False, principal=None, keytab=None, cmd_type=&apos;spark_to_jdbc&apos;, jdbc_table=None, jdbc_conn_id=&apos;jdbc-default&apos;, jdbc_driver=None, metastore_table=None, jdbc_truncate=False, save_mode=None, save_format=None, batch_size=None, fetch_size=None, num_partitions=None, partition_column=None, lower_bound=None, upper_bound=None, create_table_column_types=None, *args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.contrib.hooks.spark_submit_hook.SparkSubmitHook" title="airflow.contrib.hooks.spark_submit_hook.SparkSubmitHook"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.contrib.hooks.spark_submit_hook.SparkSubmitHook</span></code></a></p>
<p>This hook extends the SparkSubmitHook specifically for performing data
transfers to/from JDBC-based databases with Apache Spark.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first simple">
<li><strong>spark_app_name</strong> (<em>str</em>) &#x2013; Name of the job (default airflow-spark-jdbc)</li>
<li><strong>spark_conn_id</strong> (<em>str</em>) &#x2013; Connection id as configured in Airflow administration</li>
<li><strong>spark_conf</strong> (<em>dict</em>) &#x2013; Any additional Spark configuration properties</li>
<li><strong>spark_py_files</strong> (<em>str</em>) &#x2013; Additional python files used (.zip, .egg, or .py)</li>
<li><strong>spark_files</strong> (<em>str</em>) &#x2013; Additional files to upload to the container running the job</li>
<li><strong>spark_jars</strong> (<em>str</em>) &#x2013; Additional jars to upload and add to the driver and
executor classpath</li>
<li><strong>num_executors</strong> (<em>int</em>) &#x2013; number of executor to run. This should be set so as to manage
the number of connections made with the JDBC database</li>
<li><strong>executor_cores</strong> (<em>int</em>) &#x2013; Number of cores per executor</li>
<li><strong>executor_memory</strong> (<em>str</em>) &#x2013; Memory per executor (e.g. 1000M, 2G)</li>
<li><strong>driver_memory</strong> (<em>str</em>) &#x2013; Memory allocated to the driver (e.g. 1000M, 2G)</li>
<li><strong>verbose</strong> (<em>bool</em>) &#x2013; Whether to pass the verbose flag to spark-submit for debugging</li>
<li><strong>keytab</strong> (<em>str</em>) &#x2013; Full path to the file that contains the keytab</li>
<li><strong>principal</strong> (<em>str</em>) &#x2013; The name of the kerberos principal used for keytab</li>
<li><strong>cmd_type</strong> (<em>str</em>) &#x2013; Which way the data should flow. 2 possible values:
spark_to_jdbc: data written by spark from metastore to jdbc
jdbc_to_spark: data written by spark from jdbc to metastore</li>
<li><strong>jdbc_table</strong> (<em>str</em>) &#x2013; The name of the JDBC table</li>
<li><strong>jdbc_conn_id</strong> &#x2013; Connection id used for connection to JDBC database</li>
<li><strong>jdbc_driver</strong> (<em>str</em>) &#x2013; Name of the JDBC driver to use for the JDBC connection. This
driver (usually a jar) should be passed in the &#x2018;jars&#x2019; parameter</li>
<li><strong>metastore_table</strong> (<em>str</em>) &#x2013; The name of the metastore table,</li>
<li><strong>jdbc_truncate</strong> (<em>bool</em>) &#x2013; (spark_to_jdbc only) Whether or not Spark should truncate or
drop and recreate the JDBC table. This only takes effect if
&#x2018;save_mode&#x2019; is set to Overwrite. Also, if the schema is
different, Spark cannot truncate, and will drop and recreate</li>
<li><strong>save_mode</strong> (<em>str</em>) &#x2013; The Spark save-mode to use (e.g. overwrite, append, etc.)</li>
<li><strong>save_format</strong> (<em>str</em>) &#x2013; (jdbc_to_spark-only) The Spark save-format to use (e.g. parquet)</li>
<li><strong>batch_size</strong> (<em>int</em>) &#x2013; (spark_to_jdbc only) The size of the batch to insert per round
trip to the JDBC database. Defaults to 1000</li>
<li><strong>fetch_size</strong> (<em>int</em>) &#x2013; (jdbc_to_spark only) The size of the batch to fetch per round trip
from the JDBC database. Default depends on the JDBC driver</li>
<li><strong>num_partitions</strong> (<em>int</em>) &#x2013; The maximum number of partitions that can be used by Spark
simultaneously, both for spark_to_jdbc and jdbc_to_spark
operations. This will also cap the number of JDBC connections
that can be opened</li>
<li><strong>partition_column</strong> (<em>str</em>) &#x2013; (jdbc_to_spark-only) A numeric column to be used to
partition the metastore table by. If specified, you must
also specify:
num_partitions, lower_bound, upper_bound</li>
<li><strong>lower_bound</strong> (<em>int</em>) &#x2013; (jdbc_to_spark-only) Lower bound of the range of the numeric
partition column to fetch. If specified, you must also specify:
num_partitions, partition_column, upper_bound</li>
<li><strong>upper_bound</strong> (<em>int</em>) &#x2013; (jdbc_to_spark-only) Upper bound of the range of the numeric
partition column to fetch. If specified, you must also specify:
num_partitions, partition_column, lower_bound</li>
<li><strong>create_table_column_types</strong> &#x2013; (spark_to_jdbc-only) The database column data types
to use instead of the defaults, when creating the
table. Data type information should be specified in
the same format as CREATE TABLE columns syntax
(e.g: &#x201C;name CHAR(64), comments VARCHAR(1024)&#x201D;).
The specified types should be valid spark sql data
types.</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Type:</th>
<td class="field-body"><p class="first last">jdbc_conn_id: str</p>
</td>
</tr>
</tbody>
</table>




<pre>
class airflow.contrib.hooks.spark_sql_hook.SparkSqlHook(sql, conf=None, conn_id=&apos;spark_sql_default&apos;, total_executor_cores=None, executor_cores=None, executor_memory=None, keytab=None, principal=None, master=&apos;yarn&apos;, name=&apos;default-name&apos;, num_executors=None, verbose=True, yarn_queue=&apos;default&apos;)</pre>
<p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.hooks.base_hook.BaseHook</span></code></p>
<p>This hook is a wrapper around the spark-sql binary. It requires that the
&#x201C;spark-sql&#x201D; binary is in the PATH.
:param sql: The SQL query to execute
:type sql: str
:param conf: arbitrary Spark configuration property
:type conf: str (format: PROP=VALUE)
:param conn_id: connection_id string
:type conn_id: str
:param total_executor_cores: (Standalone &amp; Mesos only) Total cores for all executors</p>
<blockquote>
<div>(Default: all the available cores on the worker)</div>
</blockquote>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>executor_cores</strong> (<em>int</em>) &#x2013; (Standalone &amp; YARN only) Number of cores per
executor (Default: 2)</li>
<li><strong>executor_memory</strong> (<em>str</em>) &#x2013; Memory per executor (e.g. 1000M, 2G) (Default: 1G)</li>
<li><strong>keytab</strong> (<em>str</em>) &#x2013; Full path to the file that contains the keytab</li>
<li><strong>master</strong> (<em>str</em>) &#x2013; spark://host:port, mesos://host:port, yarn, or local</li>
<li><strong>name</strong> (<em>str</em>) &#x2013; Name of the job.</li>
<li><strong>num_executors</strong> (<em>int</em>) &#x2013; Number of executors to launch</li>
<li><strong>verbose</strong> (<em>bool</em>) &#x2013; Whether to pass the verbose flag to spark-sql</li>
<li><strong>yarn_queue</strong> (<em>str</em>) &#x2013; The YARN queue to submit to (Default: &#x201C;default&#x201D;)</li>
</ul>
</td>
</tr>
</tbody>
</table>

<pre>
run_query(cmd=&apos;&apos;, **kwargs)</pre>
<p>Remote Popen (actually execute the Spark-sql query)</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>cmd</strong> &#x2013; command to remotely execute</li>
<li><strong>kwargs</strong> &#x2013; extra arguments to Popen (see subprocess.Popen)</li>
</ul>
</td>
</tr>
</tbody>
</table>







<pre>
class airflow.contrib.hooks.spark_submit_hook.SparkSubmitHook(conf=None, conn_id=&apos;spark_default&apos;, files=None, py_files=None, driver_classpath=None, jars=None, java_class=None, packages=None, exclude_packages=None, repositories=None, total_executor_cores=None, executor_cores=None, executor_memory=None, driver_memory=None, keytab=None, principal=None, name=&apos;default-name&apos;, num_executors=None, application_args=None, env_vars=None, verbose=False)</pre>
<p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.hooks.base_hook.BaseHook</span></code>, <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.utils.log.logging_mixin.LoggingMixin</span></code></p>
<p>This hook is a wrapper around the spark-submit binary to kick off a spark-submit job.
It requires that the &#x201C;spark-submit&#x201D; binary is in the PATH or the spark_home to be
supplied.
:param conf: Arbitrary Spark configuration properties
:type conf: dict
:param conn_id: The connection id as configured in Airflow administration. When an</p>
<blockquote>
<div>invalid connection_id is supplied, it will default to yarn.</div>
</blockquote>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>files</strong> (<em>str</em>) &#x2013; Upload additional files to the executor running the job, separated by a
comma. Files will be placed in the working directory of each executor.
For example, serialized objects.</li>
<li><strong>py_files</strong> (<em>str</em>) &#x2013; Additional python files used by the job, can be .zip, .egg or .py.</li>
<li><strong>driver_classpath</strong> (<em>str</em>) &#x2013; Additional, driver-specific, classpath settings.</li>
<li><strong>jars</strong> (<em>str</em>) &#x2013; Submit additional jars to upload and place them in executor classpath.</li>
<li><strong>java_class</strong> (<em>str</em>) &#x2013; the main class of the Java application</li>
<li><strong>packages</strong> &#x2013; Comma-separated list of maven coordinates of jars to include on the</li>
</ul>
</td>
</tr>
</tbody>
</table>
<p>driver and executor classpaths
:type packages: str
:param exclude_packages: Comma-separated list of maven coordinates of jars to exclude
while resolving the dependencies provided in &#x2018;packages&#x2019;
:type exclude_packages: str
:param repositories: Comma-separated list of additional remote repositories to search
for the maven coordinates given with &#x2018;packages&#x2019;
:type repositories: str
:param total_executor_cores: (Standalone &amp; Mesos only) Total cores for all executors
(Default: all the available cores on the worker)
:type total_executor_cores: int
:param executor_cores: (Standalone, YARN and Kubernetes only) Number of cores per
executor (Default: 2)
:type executor_cores: int
:param executor_memory: Memory per executor (e.g. 1000M, 2G) (Default: 1G)
:type executor_memory: str
:param driver_memory: Memory allocated to the driver (e.g. 1000M, 2G) (Default: 1G)
:type driver_memory: str
:param keytab: Full path to the file that contains the keytab
:type keytab: str
:param principal: The name of the kerberos principal used for keytab
:type principal: str
:param name: Name of the job (default airflow-spark)
:type name: str
:param num_executors: Number of executors to launch
:type num_executors: int
:param application_args: Arguments for the application being submitted
:type application_args: list
:param env_vars: Environment variables for spark-submit. It</p>
<blockquote>
<div>supports yarn and k8s mode too.</div>
</blockquote>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><strong>verbose</strong> (<em>bool</em>) &#x2013; Whether to pass the verbose flag to spark-submit process for debugging</td>
</tr>
</tbody>
</table>

<pre>
submit(application=&apos;&apos;, **kwargs)</pre>
<p>Remote Popen to execute the spark-submit job</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>application</strong> (<em>str</em>) &#x2013; Submitted application, jar or py file</li>
<li><strong>kwargs</strong> &#x2013; extra arguments to Popen (see subprocess.Popen)</li>
</ul>
</td>
</tr>
</tbody>
</table>







<pre>
class airflow.contrib.hooks.sqoop_hook.SqoopHook(conn_id=&apos;sqoop_default&apos;, verbose=False, num_mappers=None, hcatalog_database=None, hcatalog_table=None, properties=None)</pre>
<p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.hooks.base_hook.BaseHook</span></code>, <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.utils.log.logging_mixin.LoggingMixin</span></code></p>
<p>This hook is a wrapper around the sqoop 1 binary. To be able to use the hook
it is required that &#x201C;sqoop&#x201D; is in the PATH.</p>
<p>Additional arguments that can be passed via the &#x2018;extra&#x2019; JSON field of the
sqoop connection:
* job_tracker: Job tracker local|jobtracker:port.
* namenode: Namenode.
* lib_jars: Comma separated jar files to include in the classpath.
* files: Comma separated files to be copied to the map reduce cluster.
* archives: Comma separated archives to be unarchived on the compute</p>
<blockquote>
<div>machines.</div>
</blockquote>
<ul class="simple">
<li>password_file: Path to file containing the password.</li>
</ul>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>conn_id</strong> (<em>str</em>) &#x2013; Reference to the sqoop connection.</li>
<li><strong>verbose</strong> (<em>bool</em>) &#x2013; Set sqoop to verbose.</li>
<li><strong>num_mappers</strong> (<em>int</em>) &#x2013; Number of map tasks to import in parallel.</li>
<li><strong>properties</strong> (<em>dict</em>) &#x2013; Properties to set via the -D argument</li>
</ul>
</td>
</tr>
</tbody>
</table>

<pre>
Popen(cmd, **kwargs)</pre>
<p>Remote Popen</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first simple">
<li><strong>cmd</strong> &#x2013; command to remotely execute</li>
<li><strong>kwargs</strong> &#x2013; extra arguments to Popen (see subprocess.Popen)</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th>
<td class="field-body"><p class="first last">handle to subprocess</p>
</td>
</tr>
</tbody>
</table>




<pre>
export_table(table, export_dir, input_null_string, input_null_non_string, staging_table, clear_staging_table, enclosed_by, escaped_by, input_fields_terminated_by, input_lines_terminated_by, input_optionally_enclosed_by, batch, relaxed_isolation, extra_export_options=None)</pre>
<p>Exports Hive table to remote location. Arguments are copies of direct
sqoop command line Arguments
:param table: Table remote destination
:param export_dir: Hive table to export
:param input_null_string: The string to be interpreted as null for</p>
<blockquote>
<div>string columns</div>
</blockquote>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>input_null_non_string</strong> &#x2013; The string to be interpreted as null
for non-string columns</li>
<li><strong>staging_table</strong> &#x2013; The table in which data will be staged before
being inserted into the destination table</li>
<li><strong>clear_staging_table</strong> &#x2013; Indicate that any data present in the
staging table can be deleted</li>
<li><strong>enclosed_by</strong> &#x2013; Sets a required field enclosing character</li>
<li><strong>escaped_by</strong> &#x2013; Sets the escape character</li>
<li><strong>input_fields_terminated_by</strong> &#x2013; Sets the field separator character</li>
<li><strong>input_lines_terminated_by</strong> &#x2013; Sets the end-of-line character</li>
<li><strong>input_optionally_enclosed_by</strong> &#x2013; Sets a field enclosing character</li>
<li><strong>batch</strong> &#x2013; Use batch mode for underlying statement execution</li>
<li><strong>relaxed_isolation</strong> &#x2013; Transaction isolation to read uncommitted
for the mappers</li>
<li><strong>extra_export_options</strong> &#x2013; Extra export options to pass as dict.
If a key doesn&#x2019;t have a value, just pass an empty string to it.
Don&#x2019;t include prefix of &#x2013; for sqoop options.</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
import_query(query, target_dir, append=False, file_type=&apos;text&apos;, split_by=None, direct=None, driver=None, extra_import_options=None)</pre>
<p>Imports a specific query from the rdbms to hdfs
:param query: Free format query to run
:param target_dir: HDFS destination dir
:param append: Append data to an existing dataset in HDFS
:param file_type: &#x201C;avro&#x201D;, &#x201C;sequence&#x201D;, &#x201C;text&#x201D; or &#x201C;parquet&#x201D;</p>
<blockquote>
<div>Imports data to hdfs into the specified format. Defaults to text.</div>
</blockquote>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>split_by</strong> &#x2013; Column of the table used to split work units</li>
<li><strong>direct</strong> &#x2013; Use direct import fast path</li>
<li><strong>driver</strong> &#x2013; Manually specify JDBC driver class to use</li>
<li><strong>extra_import_options</strong> &#x2013; Extra import options to pass as dict.
If a key doesn&#x2019;t have a value, just pass an empty string to it.
Don&#x2019;t include prefix of &#x2013; for sqoop options.</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
import_table(table, target_dir=None, append=False, file_type=&apos;text&apos;, columns=None, split_by=None, where=None, direct=False, driver=None, extra_import_options=None)</pre>
<p>Imports table from remote location to target dir. Arguments are
copies of direct sqoop command line arguments
:param table: Table to read
:param target_dir: HDFS destination dir
:param append: Append data to an existing dataset in HDFS
:param file_type: &#x201C;avro&#x201D;, &#x201C;sequence&#x201D;, &#x201C;text&#x201D; or &#x201C;parquet&#x201D;.</p>
<blockquote>
<div>Imports data to into the specified format. Defaults to text.</div>
</blockquote>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>columns</strong> &#x2013; &lt;col,col,col&#x2026;&gt; Columns to import from table</li>
<li><strong>split_by</strong> &#x2013; Column of the table used to split work units</li>
<li><strong>where</strong> &#x2013; WHERE clause to use during import</li>
<li><strong>direct</strong> &#x2013; Use direct connector if exists for the database</li>
<li><strong>driver</strong> &#x2013; Manually specify JDBC driver class to use</li>
<li><strong>extra_import_options</strong> &#x2013; Extra import options to pass as dict.
If a key doesn&#x2019;t have a value, just pass an empty string to it.
Don&#x2019;t include prefix of &#x2013; for sqoop options.</li>
</ul>
</td>
</tr>
</tbody>
</table>







<pre>
class airflow.contrib.hooks.ssh_hook.SSHHook(ssh_conn_id=None, remote_host=None, username=None, password=None, key_file=None, port=22, timeout=10, keepalive_interval=30)</pre>
<p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.hooks.base_hook.BaseHook</span></code>, <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.utils.log.logging_mixin.LoggingMixin</span></code></p>
<p>Hook for ssh remote execution using Paramiko.
ref: <a class="reference external" href="https://github.com/paramiko/paramiko">https://github.com/paramiko/paramiko</a>
This hook also lets you create ssh tunnel and serve as basis for SFTP file transfer</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>ssh_conn_id</strong> (<em>str</em>) &#x2013; connection id from airflow Connections from where all the required
parameters can be fetched like username, password or key_file.
Thought the priority is given to the param passed during init</li>
<li><strong>remote_host</strong> (<em>str</em>) &#x2013; remote host to connect</li>
<li><strong>username</strong> (<em>str</em>) &#x2013; username to connect to the remote_host</li>
<li><strong>password</strong> (<em>str</em>) &#x2013; password of the username to connect to the remote_host</li>
<li><strong>key_file</strong> (<em>str</em>) &#x2013; key file to use to connect to the remote_host.</li>
<li><strong>port</strong> (<em>int</em>) &#x2013; port of remote host to connect (Default is paramiko SSH_PORT)</li>
<li><strong>timeout</strong> (<em>int</em>) &#x2013; timeout for the attempt to connect to the remote_host.</li>
<li><strong>keepalive_interval</strong> (<em>int</em>) &#x2013; send a keepalive packet to remote host every
keepalive_interval seconds</li>
</ul>
</td>
</tr>
</tbody>
</table>

<pre>
create_tunnel(**kwds)</pre>
<p>Creates a tunnel between two hosts. Like ssh -L &lt;LOCAL_PORT&gt;:host:&lt;REMOTE_PORT&gt;.
Remember to close() the returned &#x201C;tunnel&#x201D; object in order to clean up
after yourself when you are done with the tunnel.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first simple">
<li><strong>local_port</strong> (<em>int</em>) &#x2013; </li>
<li><strong>remote_port</strong> (<em>int</em>) &#x2013; </li>
<li><strong>remote_host</strong> (<em>str</em>) &#x2013; </li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th>
<td class="field-body"><p class="first last"></p>
</td>
</tr>
</tbody>
</table>







<pre>
class airflow.contrib.hooks.vertica_hook.VerticaHook(*args, **kwargs)</pre>
<p>Bases: <a class="reference internal" href="#airflow.hooks.dbapi_hook.DbApiHook" title="airflow.hooks.dbapi_hook.DbApiHook"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.hooks.dbapi_hook.DbApiHook</span></code></a></p>
<p>Interact with Vertica.</p>

<pre>
get_conn()</pre>
<p>Returns verticaql connection object</p>







<pre>
class airflow.contrib.hooks.wasb_hook.WasbHook(wasb_conn_id=&apos;wasb_default&apos;)</pre>
<p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.hooks.base_hook.BaseHook</span></code></p>
<p>Interacts with Azure Blob Storage through the wasb:// protocol.</p>
<p>Additional options passed in the &#x2018;extra&#x2019; field of the connection will be
passed to the <cite>BlockBlockService()</cite> constructor. For example, authenticate
using a SAS token by adding {&#x201C;sas_token&#x201D;: &#x201C;YOUR_TOKEN&#x201D;}.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><strong>wasb_conn_id</strong> (<em>str</em>) &#x2013; Reference to the wasb connection.</td>
</tr>
</tbody>
</table>

<pre>
check_for_blob(container_name, blob_name, **kwargs)</pre>
<p>Check if a blob exists on Azure Blob Storage.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first simple">
<li><strong>container_name</strong> (<em>str</em>) &#x2013; Name of the container.</li>
<li><strong>blob_name</strong> (<em>str</em>) &#x2013; Name of the blob.</li>
<li><strong>kwargs</strong> (<em>object</em>) &#x2013; Optional keyword arguments that
<cite>BlockBlobService.exists()</cite> takes.</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th>
<td class="field-body"><p class="first last">True if the blob exists, False otherwise.</p>
</td>
</tr>
</tbody>
</table>
<p>:rtype bool</p>




<pre>
check_for_prefix(container_name, prefix, **kwargs)</pre>
<p>Check if a prefix exists on Azure Blob storage.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first simple">
<li><strong>container_name</strong> (<em>str</em>) &#x2013; Name of the container.</li>
<li><strong>prefix</strong> (<em>str</em>) &#x2013; Prefix of the blob.</li>
<li><strong>kwargs</strong> (<em>object</em>) &#x2013; Optional keyword arguments that
<cite>BlockBlobService.list_blobs()</cite> takes.</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th>
<td class="field-body"><p class="first last">True if blobs matching the prefix exist, False otherwise.</p>
</td>
</tr>
</tbody>
</table>
<p>:rtype bool</p>




<pre>
get_conn()</pre>
<p>Return the BlockBlobService object.</p>




<pre>
get_file(file_path, container_name, blob_name, **kwargs)</pre>
<p>Download a file from Azure Blob Storage.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>file_path</strong> (<em>str</em>) &#x2013; Path to the file to download.</li>
<li><strong>container_name</strong> (<em>str</em>) &#x2013; Name of the container.</li>
<li><strong>blob_name</strong> (<em>str</em>) &#x2013; Name of the blob.</li>
<li><strong>kwargs</strong> (<em>object</em>) &#x2013; Optional keyword arguments that
<cite>BlockBlobService.create_blob_from_path()</cite> takes.</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
load_file(file_path, container_name, blob_name, **kwargs)</pre>
<p>Upload a file to Azure Blob Storage.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>file_path</strong> (<em>str</em>) &#x2013; Path to the file to load.</li>
<li><strong>container_name</strong> (<em>str</em>) &#x2013; Name of the container.</li>
<li><strong>blob_name</strong> (<em>str</em>) &#x2013; Name of the blob.</li>
<li><strong>kwargs</strong> (<em>object</em>) &#x2013; Optional keyword arguments that
<cite>BlockBlobService.create_blob_from_path()</cite> takes.</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
load_string(string_data, container_name, blob_name, **kwargs)</pre>
<p>Upload a string to Azure Blob Storage.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>string_data</strong> (<em>str</em>) &#x2013; String to load.</li>
<li><strong>container_name</strong> (<em>str</em>) &#x2013; Name of the container.</li>
<li><strong>blob_name</strong> (<em>str</em>) &#x2013; Name of the blob.</li>
<li><strong>kwargs</strong> (<em>object</em>) &#x2013; Optional keyword arguments that
<cite>BlockBlobService.create_blob_from_text()</cite> takes.</li>
</ul>
</td>
</tr>
</tbody>
</table>




<pre>
read_file(container_name, blob_name, **kwargs)</pre>
<p>Read a file from Azure Blob Storage and return as a string.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>container_name</strong> (<em>str</em>) &#x2013; Name of the container.</li>
<li><strong>blob_name</strong> (<em>str</em>) &#x2013; Name of the blob.</li>
<li><strong>kwargs</strong> (<em>object</em>) &#x2013; Optional keyword arguments that
<cite>BlockBlobService.create_blob_from_path()</cite> takes.</li>
</ul>
</td>
</tr>
</tbody>
</table>







<pre>
class airflow.contrib.hooks.winrm_hook.WinRMHook(ssh_conn_id=None, remote_host=None, username=None, password=None, key_file=None, timeout=10, keepalive_interval=30)</pre>
<p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.hooks.base_hook.BaseHook</span></code>, <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.utils.log.logging_mixin.LoggingMixin</span></code></p>
<p>Hook for winrm remote execution using pywinrm.</p>
<table class="docutils field-list" frame="void" rules="none">
<colgroup><col class="field-name">
<col class="field-body">
</colgroup>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th>
<td class="field-body"><ul class="first last simple">
<li><strong>ssh_conn_id</strong> (<em>str</em>) &#x2013; connection id from airflow Connections from where all
the required parameters can be fetched like username, password or key_file.
Thought the priority is given to the param passed during init</li>
<li><strong>remote_host</strong> (<em>str</em>) &#x2013; remote host to connect</li>
<li><strong>username</strong> (<em>str</em>) &#x2013; username to connect to the remote_host</li>
<li><strong>password</strong> (<em>str</em>) &#x2013; password of the username to connect to the remote_host</li>
<li><strong>key_file</strong> (<em>str</em>) &#x2013; key file to use to connect to the remote_host.</li>
<li><strong>timeout</strong> (<em>int</em>) &#x2013; timeout for the attempt to connect to the remote_host.</li>
<li><strong>keepalive_interval</strong> (<em>int</em>) &#x2013; send a keepalive packet to remote host
every keepalive_interval seconds</li>
</ul>
</td>
</tr>
</tbody>
</table>



</div>
</div>
<div class="section" id="executors">
<h2 class="sigil_not_in_toc">Executors</h2>
<p>Executors are the mechanism by which task instances get run.</p>

<pre>
class airflow.executors.local_executor.LocalExecutor(parallelism=32)</pre>
<p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.executors.base_executor.BaseExecutor</span></code></p>
<p>LocalExecutor executes tasks locally in parallel. It uses the
multiprocessing Python library and queues to parallelize the execution
of tasks.</p>

<pre>
end()</pre>
<p>This method is called when the caller is done submitting job and is
wants to wait synchronously for the job submitted previously to be
all done.</p>




<pre>
execute_async(key, command, queue=None, executor_config=None)</pre>
<p>This method will execute the command asynchronously.</p>




<pre>
start()</pre>
<p>Executors may need to get things started. For example LocalExecutor
starts N workers.</p>




<pre>
sync()</pre>
<p>Sync will get called periodically by the heartbeat method.
Executors should override this to perform gather statuses.</p>







<pre>
class airflow.executors.celery_executor.CeleryExecutor(parallelism=32)</pre>
<p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.executors.base_executor.BaseExecutor</span></code></p>
<p>CeleryExecutor is recommended for production use of Airflow. It allows
distributing the execution of task instances to multiple worker nodes.</p>
<p>Celery is a simple, flexible and reliable distributed system to process
vast amounts of messages, while providing operations with the tools
required to maintain such a system.</p>

<pre>
end(synchronous=False)</pre>
<p>This method is called when the caller is done submitting job and is
wants to wait synchronously for the job submitted previously to be
all done.</p>




<pre>
execute_async(key, command, queue=&apos;default&apos;, executor_config=None)</pre>
<p>This method will execute the command asynchronously.</p>




<pre>
start()</pre>
<p>Executors may need to get things started. For example LocalExecutor
starts N workers.</p>




<pre>
sync()</pre>
<p>Sync will get called periodically by the heartbeat method.
Executors should override this to perform gather statuses.</p>







<pre>
class airflow.executors.sequential_executor.SequentialExecutor</pre>
<p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.executors.base_executor.BaseExecutor</span></code></p>
<p>This executor will only run one task instance at a time, can be used
for debugging. It is also the only executor that can be used with sqlite
since sqlite doesn&#x2019;t support multiple connections.</p>
<p>Since we want airflow to work out of the box, it defaults to this
SequentialExecutor alongside sqlite as you first install it.</p>

<pre>
end()</pre>
<p>This method is called when the caller is done submitting job and is
wants to wait synchronously for the job submitted previously to be
all done.</p>




<pre>
execute_async(key, command, queue=None, executor_config=None)</pre>
<p>This method will execute the command asynchronously.</p>




<pre>
sync()</pre>
<p>Sync will get called periodically by the heartbeat method.
Executors should override this to perform gather statuses.</p>






<div class="section" id="community-contributed-executors">
<h3 class="sigil_not_in_toc">Community-contributed executors</h3>

<pre>
class airflow.contrib.executors.mesos_executor.MesosExecutor(parallelism=32)</pre>
<p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.executors.base_executor.BaseExecutor</span></code>, <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.www.utils.LoginMixin</span></code></p>
<p>MesosExecutor allows distributing the execution of task
instances to multiple mesos workers.</p>
<p>Apache Mesos is a distributed systems kernel which abstracts
CPU, memory, storage, and other compute resources away from
machines (physical or virtual), enabling fault-tolerant and
elastic distributed systems to easily be built and run effectively.
See <a class="reference external" href="http://mesos.apache.org/">http://mesos.apache.org/</a></p>

<pre>
end()</pre>
<p>This method is called when the caller is done submitting job and is
wants to wait synchronously for the job submitted previously to be
all done.</p>




<pre>
execute_async(key, command, queue=None, executor_config=None)</pre>
<p>This method will execute the command asynchronously.</p>




<pre>
start()</pre>
<p>Executors may need to get things started. For example LocalExecutor
starts N workers.</p>




<pre>
sync()</pre>
<p>Sync will get called periodically by the heartbeat method.
Executors should override this to perform gather statuses.</p>






</div>
</div>
</body>
</html>